Method and system for processing data on motion pictures by motion compensation combined with image segmentation

ABSTRACT

Measures for a motion compensation are combined with an image segmentation in four modes. In a first mode, a current picture is image-segmented before an estimation of a relative motion between a segmented region of the current picture and a corresponding region of a reference picture. In a second mode, a relative motion is estimated between a minute piece of a current picture and a corresponding minute piece of a reference picture, to be employed as a character parameter for an image segmentation. In a third mode, a relative motion is estimated between a sub-region of an image-segmented region and a corresponding small region, to be employed as a character parameter for a resegmentation. In a fourth mode, a relative motion between minute pieces is estimated to provide an additional character parameter for an image segmentation to be performed on a current picture, before a reestimation of a relative motion between an estimated region and a corresponding region.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to a method and a system forprocessing a set of data on motion pictures, and particularly to amethod and a system for processing a set of data on a temporal sequenceof two or three dimensional pictures with a time-dependent motionintermittently picked up therein, by using measures for a compensationof the motion in combination with an image segmentation of the pictures.

2. Description of the Related Art

Recent years have observed an increasing interest in a processing of aset of digital data on motion pictures, in particular in a field of artrelating to a compression coding of such data.

The motion pictures appear as a temporal sequence of pictures eachhaving an image region associated with a transient state of a motion.The transient state in an arbitrary picture has a mathematicallyanalyzable correlation to that in a previous picture, so that the formeris predictable from the latter in combination with an estimatedcorrelation therebetween or by compensating the latter therewith,subject to an adequacy of an employed model of the motion for analysingthe correlation.

A predicted current state is comparable with a sampled current state todetermine a difference therebetween as a prediction error, which isencodable together with associated parameter values of the model toobtain a compressed code, which permits a current state of the motion tobe calculated in combination with a past calculated state thereof at adecoding end.

An adequate model provides an effective prediction to achieve asignificant reduction in redundancy of the code.

A typical modelling is based on a motion compensation interframeprediction, in which a motion is defined in terms of a displacement of ablock of a picked up image between a pair of frames each correspondingto a picture. Typically, the picture is divided into a set of squareblocks each having an area about an order of 16×16 pixels. A totalnumber of blocks in any picture is identical for this purpose, andrespective blocks in an arbitrary picture one-to-one correspond to thosein any other one. With respect to each block, a past decoded picture iscompensated by an estimated motion, to predict a picture to be comparedwith a current input picture. Thus, motion compensation is made inblocks, together with associated pixel data kept as they are inherent.

FIG. 1 exemplarily shows an encoder of a conventional motion picturecoding-decoding system using the motion compensation interframeprediction. This conventional system is well known by the ITU-T(CCITT)Recommendations H. 261, as a video coded for audiovisual services atp×64 kbit/s.

In FIG. 1, designated at reference character 100 is an entirety of theconventional system, 100a is the encoder, 101 is an input terminal ofthe encoder 100a, and 102 and 109 are frame memories, respectively.

The input terminal 101 inputs a sequence of pixel data D101 of a currentpicture P101. The frame memory 102 stores therein the data D101 of thecurrent input picture P101, besides a set of data D100 of a past inputpicture P100 it received from the input terminal 101 and stored thereinin a last frame. The frame memory 109 has stored therein a set of dataD102 of a local decoded picture P102 that is a matrix of restored pixeldata of the past input picture P100.

The data D100 of the past input picture P100 are sequentially read fromthe frame memory 102, and the data D102 of the local decoded pictureP102 from the frame memory 109. They are either selected by a switch103a, as a sequence of data D103 representing a reference picture P103(to be P100 or P102), and input to a motion estimator 103, whichconcurrently receives the data D101 of the current input picture P101.

Incidentally, as shown in FIG. 2, the conventional system 100 has, as acommon field Fc to a variety of associated computations, an imaginaryorthogonal coordinate system defined by a combination of an axis ofabscissa X corresponding to a bottom side of a picture frame Fp and anaxis of ordinate Y corresponding to a left lateral side of the pictureframe Fp. The picture frame Fp, as well as any block B (β/α) therein anda geometrical gravity center G (β/α) thereof, is congruently mapped inthe imaginary field Fc each time when a set of associated data isprocessed in the system 100, where "α" is a picture identificationnumber and "β" is a block identification number.

As illustrated in FIG. 2, the motion estimator 103 estimates, bycalculation for each block B(i/101) (i=an arbitary integer) of thecurrent input picture P101, a motion as a displacement vector Vd_(i) interms of a combination of a sense and a magnitude of a displacement of agravity center G(i/101) of an i-th block B(i/101) in the current inputpicture P101 relative to that G(i/103) of a corresponding i-th blockB(i/103) in the reference picture P103, thereby obtaining a set of dataD104 of a set Vm of motion vectors of which an i-th one Vm_(i) consistsof an X-component Vx_(i) as a projection of the displacement vectorVd_(i) to the axis X and a Y-component Vy_(i) as that to the axis Y.

The data D104 of the motion vector set Vm are sequentially output fromthe motion estimator 103 to a motion compensator 104 and an encodingmultiplexer 110.

The multiplexer 110 encodes the data D104 of the vector set Vm into asequence of corresponding codes.

At the motion compensator 104, the data D104 of the motion vector set Vmare processed together with the data D102 of the local decoded pictureP102 input from the frame memory 109 so that a gravity center B(i/102)of each block B(i/102) in the local decoded picture P102 has a positionthereof displaced or coordinate-componentwise compensated along the axesX and Y by equivalent distances to X- and Y-components Vx_(i) and Vy_(i)of a corresponding motion vector Vm_(i) respectively, to thereby obtaina motion-compensated picture P104 as an interframe predicted one for thecurrent input picture P101.

As a result, each pixel data in each displaced block in themotion-compensated picture P104 is updated by a data that acorresponding pixel in that block in the local decoded picture P102 hadbeen carrying.

The motion compensator 104 sequentially outputs a set of data D105 ofthe motion-compensated picture P104 to a subtractor 105 and an adder108.

The subtractor 105 performs a pixel-mode subtraction of themotion-compensated picture P104 from the current input picture P101,obtaining a set of data D106 representative of a differential pictureP105 therebetween, i.e. a picture having elementwise distributed thereona matrix of prediction errors due to a motion compensation in a16×16-pixel block mode by the compensator 104. Accordingly, the dataD101 of the current input picture P101 are converted into a compressedset of data as the data D106 respresentative the prediction errors.

The prediction error data D106 of the differential picture P105 aresequentially input from the subtractor 105 to a unit 106 adapted for adiscrete cosine transformation and quantization (hereafter "DCT-Q")process, i.e. for a data compression, where they are mapped in an8×8-pixel block mode from a real measure space through a discrete cosinetransform function into a related frequency field, to be expressed interms of a combination of cosine coefficients of a corresponding cosineseries to an associated 8×8-pixel block, and the cosine coefficients arequantized.

As a result, the error data D106 are further compressed into a set ofcombinations of data D107 each representative of a quantizedcoefficient, so that the set of data D107 represents the differentialpicture P105.

The compressed data D107 of the differential picture P105 aresequentially output from the DCT-Q unit 106 to the encoding multiplexer110 and a unit 107 adapted for an inverse quantization and inversediscrete cosine transformation (hereafter "IQ-IDCT") process, i.e. for adata decompression.

The multiplexer 110 encodes the compressed data D107 of the differentialpicture P105 into a sequence of corresponding codes, and multiplexesthem together with the codes of the the motion vector set Vm, into asequence of multiplexed codes C101 to be output via an output terminal111 of the encoder 100a.

At the IQ-IDCT unit 107, each input combination of data D107 is inversequantized into a combination of corresponding cosine coefficients of acosine series in a related frequency field, which coefficients are theninverse mapped from the frequency field through an inverse discretecosine transform function into the real measure space, so that acorresponding 8×8-pixel block in a vacant picture frame Fp mapped in theX-Y coordinate system has pixel data thereof equivalent to correspondingones in the differential picture P105.

In due course, the picture frame Fp becomes solid with a set of suchpixel data D109, thus resulting in a restored differential picture P106equivalent to the differential picture P105.

The data D109 of the restored differential picture P106 are sequentiallyoutput to the adder 108, where they are subjected to a pixel-modeaddition with the data D105 of the motion-compensated picture P104 inputfrom the motion compensator 104, to thereby obtain a set of data D110representative of a local restored picture P107 equivalent to thecurrent input picture P101.

The data D110 of the restored current picture P107 is sequentially inputfrom the adder 108 to the frame memory 109, where they are stored atcorresponding addressed locations, as a set of data representing a localdecoded current picture to be employed, in a subsequent frame, as asubsequent local decoded picture of a subsequent past input picture.

Incidentally, the sequence of output codes C101 is transmitted through atransmission line 112 to a decoder side of the system 100.

FIG. 3 exemplarily shows a decoder of the conventional motion picturecoding-decoding system 100.

In FIG. 3, designated at reference character 100b is the decoder of thesystem 100, 120 is a decoding demultiplexer, 121 is an input terminal ofthe decoder 100b, and 125 is a frame memory. The decoding demultiplexer120 receives the multiplexed codes C110 representative of thedifferential picture P105 and the motion vector set Vm via the inputterminal 121 connected to the transmission line 112. The frame memoryhas stored therein a set of data D120 representative of a past decodedpicture P120 equivalent to the local decoded picture P102 in the encoder100a.

The demultiplexer 120 demultiplexes the codes C110 into a code sequencerepresentative of the differential picture P105 and a code sequencerepresentative of the vector set Vm, and decodes the former into asequence of data D121 equivalent to the compressed data D107 in theencoder 100a and the latter into a sequence of data D122 equivalent tothe data D104 in the encoder 100a.

The data D121 are input to an IQ-IDCT unit 122, which functions in asimilar manner to the IQ-IDCT unit 107 in the encoder 100a, thussequentially outputting a set of data D123 representative of adifferential picture P121 equivalent to the restored differentialpicture P106.

The data D122 are input to a motion compensator 123, which concurrentlyreceives the data D120 of the past decoded picture P120 from the framememory 125 and compensates these data D120 by those data D122 in asimilar manner to the motion compensator 104 in the encoder 100a, thussequentially outputting a set of data D124 representative of amotion-compensated picture P122 as a predicted current pictureequivalent to the motion-compensated picture P104.

The data D123 of the differential picture P121 and the data D124 of themotion-compensated picture P122 are input to an adder 124, where theyare added to each other in a similar manner to the adder 108 in theencoder 100a, to thereby obtain a set of data D125 representative of acurrent decoded picture P123 equivalent to the local decoded pictureP107, which data are output as a datastream via an output terminal 126of the decoder 100b. This datastream is branched to be input to theframe memory 125, where it is stored as a set of data reprentative ofthe current decoded picture P123 to be employed as a subsequent pastdecoded picture in the subsequent frame.

The motion compensation interframe prediction permits an effective datacompression of a motion picture even in the conventional coding-decodingsystem 100.

In the conventional system 100, however, a single motion is estimated todetermine a single vector for each of all square blocks having a16×16-pixel size. Such a restriction constitutes some drawbacks.

For example, in a block, some pixels may represent an image of an objectmoving in a certain direction, and others, that of another object movingin a different direction. An estimated motion of such a block mayprovide an erroneous motion vector, causing an associated predictionerror to be increased, thus resulting in a reduced coding efficiency.

Still less, in a picture, some block may represent an image of a certainportion of a moving object, and another, that of a neighboring portionthereof. An estimated motion of the former may be different from that ofthe latter, giving rise to an errorneous contour or discontinuity acrossa continuous image region of the object, thus resulting in a degradedpicture quality.

Recent years have further observed an increasing interest in aprocessing of data on motion pictures, in relation to an imagesegmentation.

The image segmentation is a growing technique for reducing a redundancyin a stream of data on a set of pictures that may be a sequence ofcolored motion pictures each consisting of a matrix of picture elementsor pixels.

An arbitrary pixel Px in such a picture sequence is identifiable by alocation Lc thereof in an associated picture Pc identified by a framenumber Nf thereof, such that Px=Px (Lc; Pc(Nf))=Px(Lc; Nf). The locationLc may be defined by a combination (x,y) of coordinates x and y in anx-y coordinate system fixed to the picture Pc or by an address definedin a pixel matrix. The frame number Nf may be defined by a measure tfrom an initial time t₀ on a real time axis, such that Nf=Nf(t)=Gs(t=t₀)/Tf , where Tf is a frame period and Gs is a Gauss step function.

In general, the pixel Px(Lc; Nf) is characterized by characterinformation or data associated therewith in terms of a set φ ofcharacter parameters φ_(a) such as a combination of variantsrepresentative of coordinates in an R-G-B color coordinate system and/oran associated luminance, where "a" is an arbitrary one of a plurality ofcharacter identification numbers.

Accordingly, an arbitrary pixel Px in an arbitrary picture Pc may bedefined such that Px=Px(Lc, φ; Nf)=Px(x, y, {φ_(a) }; t). Thus, lettingx and y also be character parameters φ_(x) and φ_(y), respectively, thepixel Px may be defined by Px=Px(φ'; t), where φ' is an extendedcharacter parameter set such that φ'={φ_(x), φ_(y) {φ_(a) } }={φ_(b) },where "b" is an extended character identification number so that "b" maybe "x", "y" or "a".

At a particular time point t=t_(c), therefore, an associated picture Pcmay be defined such that Pc=Ψ, where Ψ is a union of extended characterparameter sets φ' of the picture Pc so that Ψ={∪φ'; t=t_(c) (i.e.parameters φ_(b) in each set φ' are valued)}.

The image segmentation dissolves the union set Ψ of the valued parametersets φ' (at t=t_(c)) to a predetermined number of subsets Ψ_(d)(d=subset identification number) thereof each consisting of a variablenumber of parameter sets φ', so that no parameter set φ' is sharedbetween any pair of subsets Ψ_(d) and that each parameter φ_(b) has anarbitrary pair of values thereof both in a subset Ψ_(d), as they arealike or relatively vicinal to each other, and either in both of a pairof subsets Ψ_(d), as they are relatively distant from each other.

The dissolving may comprise a clustering, as discussed in a paper"Combining Color and Spatial Information for Segmentation" by NobuakiIZUMI et al., the Proceedings of the 1991 Spring Term NationalConference of the Institute of the Electronics, Intelligence andCommunication Engineers of Japan, D680, p. 7-392.

FIG. 4 illustrates a basic concept of the clustering, as it has anexemplarily reduced number of parameter dimensions to permit anintuitive comprehension. Like items to FIG. 2 are designated by likereference characters. For brevity, notations of parameter sets orelements thereof will be commonly applied to all spaces, permittingscales of associated coordinate axes to be varied, providing that x=Xand y=Y.

As shown at the left side of FIG. 4, a common field Fc defined in animaginary X-Y coordinate system has mapped therein from an unshown realspace a matrix as a union set Ψ of valued parameter sets φ' eachconsisting of five character parameters φ_(x) =X(λ/α), φ_(y) =Y(λ/α), φ₂=R(λ/α) (data on a red color), φ₃ =G(λ/α) (data on a green color) and φ₄=B(λ/α) (data on a blue color) and of a corresponding pixel Px(λ/α) in apicture Pc with an identification number α (i.e. t=t_(c) =t₀ +Tf×α),where "λ" is a pixel identification number.

In the common field Fc, therefore, a set φ of color parameters φ_(a)(a=2, 3, 4) of each pixel Px(λ/α) is degenerated at a correspondingcoordinate (X,Y). In place of the color parameters, there may beemployed a set of luminance and chrominance parameters valued as Y(λ/α),Cb(λ/α) and Cr(λ/α).

The parameter values R(λ/α), G(λ/α) and B(λ/α) of any pixel Px(λ/α)represent in combination a chromatic color that is identical or vicinalto, for example, a first color illustrated by a white circle, a secondcolor illustrated by a shadowed circle or a third color illustrated by ablack circle.

As shown in mid of FIG. 4, the union set Ψ={φ'} is elementwise mappedinto a three-dimensional parameter space defined by an R-G-B coordinatesystem with an R-axis, a G-axis and a B-axis representative of characterparameters φ₂, φ₃ and φ₄, respectively, while the remaining parametersφ_(x) and φ_(y) are degenerated therein.

Accordingly, those pixels illustrated by the white circle are all mappedin a connected region 1, as their colors are identical or vicinal toeach other. They constitute a set of vicinal points called "cluster (inthe parameter space)" corresponding to a subset Ψ_(d) which also iscalled "cluster (in the common field Fc)". Each pixel-representativepoint (R, G, B) in the cluster is labelled as a element thereof, with acluster identification number equivalent to the subset identificationnumber d (to be 1 in this case).

Likewise, those illustrated by the shadowed circle are mapped to beclustered in a connected region 2, and those illustrated by the blackcircle in a connected region 3.

There are thus constituted three clusters 1, 2 and 3 in the regions 1, 2and 3 disconnected from each other in the parameter space.

Then, as shown at the right side of FIG. 4, the clusters 1, 2 and 3 areelementwise inverse mapped into the common field Fc, so that theirelements are each mapped in a form of a pixel Px(λ/α) with acorresponding label d (1, 2 or 3) representative of a cluster (as thesubset Ψ_(d)) it belongs in this field Fc.

As a result, an image segmentation is performed by a clustering.

In the case of FIG. 4, however, the parameter space has two degeneratedparameters φ_(x) and φ_(y) for the convenience of illustration, so thatthe clustering is performed of three parameters φ₂, φ₃ and φ₄, thusresulting in three clusters 1, 2 and 3 separated from each other by adashed line in the field Fc. The cluster 1 in Fc is provided as acombination of a right lower region and a left upper region distant fromeach other.

In a practical clustering, therefore, there is employed afive-dimensional parameter space including X and Y axes for theparameters φ_(x) and φ_(y), so that the right lower and left upperregions of the cluster 1 may have a significant distance detectedtherebetween in the parameter space and may thus be clustered in thefield Fc either with a label 1 and the other with a label 4.

Another practical clustering may employ a common field Fc with a set ofluminance and chrominance parameters φ_(a), i.e. a five-dimensionalparameter space including Y, Cb and Cr axes for the parameters φ_(a).

Incidentally, for a comprehensive classification of various fields andspaces, there will sometimes be employed herein three notations "S"representing a real or imaginary spatial field, "ST" representing a realor imaginary spatiotemporal field, and "[e]" representing a measure "e"of dimension, where "e" is an arbitrary integer. For example, the commonfield Fc is an ST[3] class, and the parameter space defined by the R-G-Bcoordinate system is an S[3] class.

FIG. 5 shows an exemplary motion picture processing system fordescribing a conventional image segmentation using a five-dimensionalclustering.

In FIG. 5, designated at reference character 200 is an entirety of theprocessing system. Like terms will be designated by like charactersbetween the foregoing description and the following description.

The system 200 comprises a frame memory 210, an address generator 220and a clustering circuit 230.

The frame memory 210 receives for storing therein a triple of sequencesof parallel color data D200 of a current input picture P200 from anunshown input port, in synchronism with a sequence of write address dataD221w output thereto from the address generator 220, and outputstherefrom a triple of sequences of parallel color data D210 of a certainpicture P210 in concern (that may be a past input picture or the currentinput picture), in synchronism with a sequence of read address dataD221r output thereto from the address generator 220. The picture P210 isnow assumed to be equivalent to the picture Pc at t=t_(c) in FIG. 4, foran intuitive comprehension.

The frame memory 210 is composed of a triple of parallel memories, i.e.an R-memory 211, a G-memory 212 and a B-memory 213.

The color data D200 include a set of R-data on a red color, a set ofG-data on a green color and a set of B-data on a blue color respectivelyof the picture P200. The color data D210 also include a set {φ₂ } ofR-data D211=R(λ/α) on a red color, a set {φ₃ } of G-data D212=G(λ/α) ona green color and a set {φ₄ } of B-data D213=B(λ/α) on a blue colorrespectively of the picture P210.

The write address data D221w as well as the read address data D221r eachconsist of a pair of address data. The address data pair in the dataD221w defines a write location in each of the R-, G- and B memories 211,212 and 213, which location corresponds to a pixel position in thepicture P200. The address data pair in the data D221r defines a readlocation in each of the R-, G- and B memories 211, 212 and 213, whichlocation corresponds to a two-dimensional coordinate (x,y) of a positionthat a pixel Px has in the picture P210.

The R-, G- and B-data of the picture P200 as input to the frame memory210 are written in the R-, G- and B-memories 211, 212 and 213,respectively, at write locations therein defined by the address dataD221w. The R-, G and B-data of the picture P210 to be output from theframe memory 210 are read from the R-, G- and B-memories 211, 212 and213, respectively, and more specifically, from read locations thereindefined by the address data D221r.

The address generator 220 further outputs a sequence of location dataD222 in synchronism with the read address data D221r. Each location dataD222 represents (or consists of) a corresponding one of the read addressdata D221r, and comprises a pair of data D223 and D224 representing,either D223, an x-coordinate φ_(x) and, the other D224, a y-coordinateφ_(y) respectively of the afore-mentioned coordinate (x,y).

The clustering circuit 230 processes a synchronized combination {φ'} ofthe color data D210={₂, φ₃, φ₄ } of the picture P210 and the locationdata D222={φ_(x), φ_(y) } so that, in a frame Nf (=α) of time, thepicture P210 as a union set Ψ of valued character parameter sets φ' iscongruently mapped in a common field Fc of an ST[3] class, wherefrom theunion set Ψ={X(λ/α), Y(λ/α), R(λ/α), G(λ/α), B(λ/α) } is mapped to apractical five-dimensional parameter space of an S[5] class consistingof a set of spatial points each defined by a coordinate (X,Y,R,G,B) orby a coordinate (R,G,B,X,Y), while the latter is employed in this casefor an intuitive consistency with FIG. 4.

In the parameter space, the union set Ψ is elementwise clustered in aset {Ci} of n clusters Ci, where "n" is a predetermined arbitraryinteger and "i" is an arbitrary integer such that 1≦i≦n. In this case,{Ci}={1, 2, 3, 4}={C₁, C₂, C₃, C₄ } and n=4.

Then, the n clusters Ci[5] in the S[5] field are inverse mapped to theST[3] field Fc. As a result, in the frame α, the picture Pc as atwo-dimensional set of pixels Px(λ/α) in this field Fc is grouped orclustered into n clusters Ci[2] in terms of valued subsets Ψ_(d)labelled 1 to 4 in this case, i.e., it is apparently segmented into nregions RS each connected therein and labelled with a number "i"(corresponding to i of Ci and d of Ψ_(d)) such that RSi.

The clustering circuit 230 sequentially outputs a set of data D230 onsuch the result of image segmentation, each including clock-dependent orvalued information on a combination of a pixel identification number λand an associated label i as a cluster or region identification number.

The clustering and hence the segmentation is a due result of a mappingto or from an imaginary space, based on a real computation according toan algorism.

The algorism will be described below for an arbitrary integer n (notlimited to 4), providing that a number of scales are determined as colorand location parameter weighting factors k₀ and k₁ in the mappingbetween the ST[3] class field Fc and the prameter field of S[5] class,such that:

    R[5]=k.sub.0.sup.1/2 ×R[2]                           (1),

    G[5]=k.sub.0.sup.1/2 ×G[2]                           (2),

    B[5]=k.sub.0.sup.1/2 ×B[2]                           (3),

    X[5]=k.sub.1.sup.1/2 ×X[2]                           (4),

and

    Y[5]=k.sub.1.sup.1/2 ×Y[2]                           (5).

In the conventional system 200, the picture Pc is initially equi-dividedinto n square blocks Ci₀ each having a geometrically central pixelPg_(i0) therein as a representative pixel Pr_(i0) thereof with a set φ'of associated parameter values componentwise representative of afive-dimensional parameter vector Vp_(i0) defined in the S[5] field,such that: ##EQU1##

Thus, each block C_(i0) is represented by the vector Vp_(i0) as arepresentative vector thereof in the S[5] field.

Likewise, every pixel Px(λ(x,y)/α) in the picture Pc is represented inthe S[5] field by a corresponding five-dimensional parameter vectorVp_(xy), such that: ##EQU2##

Then, each parameter vector VP_(xy) has a euclidean relative distanceD_(io) thereof determined to the representative vector Vp_(i0) of eachblock C_(i0), such that: ##EQU3##

Each pixel Px(λ(x,y)/α) represented by the parameter vector Vp_(xy) thushas n relative distances D_(i0) determined therefor, including a minimumone D_(min-0), and is labelled with an identification number i₀ (or anincremented number i₁) of a block C_(i0) associated with the minimumdistance D_(min-0).

Accordingly, all pixels Px of the picture Pc in the field Fc are eachlabelled with a corresponding one of n identification numbers i₀ (or i₁)(i=1 to n), so that the picture Pc is re-segmented into n first-orderclusters C_(i1) (i=1 to n) each consisting of N_(i1) pixels Px_(i1)labelled with an identical number i₀ (or i₁), while the number N_(i1) ofpixels Px_(i1) is variable.

The N_(i1) pixels Px_(i1) in each first-order cluster C_(i1) have theirparameter values R(λ(x,y)/α), G(λ(x,y)/α), B(λ(x,y)/α), X(λ(x,y)/α) andY(λ(x,y)/α) arithmetically averaged thereamong to obtain a set ofrepresentative parameter values R_(ci1), G_(ci1), B_(ci1), X_(ci1) andY_(ci1) of the first-order cluster C_(i1), such that:

    R.sub.ci1 =(ΣR(λ(x,y)/α))/N.sub.i1      (9),

    G.sub.ci1 =(ΣG(λ(x,y)/α))/N.sub.i1      (10),

    B.sub.ci1 =(ΣB(λ(x,y)/α))/N.sub.i1      (11),

    X.sub.ci1 =(ΣX(λ(x y)/α))/N.sub.i1      (12),

and

    Y.sub.ci1 =(ΣY(λ(x,y)/α))/N.sub.i1      (13),

where the sum Σ is taken of the N_(i1) pixels Px_(i1).

The cluster C_(i1) is represented by a representative vector Vp_(i1)defined in the S[5] field, such that:

    Vp.sub.i1 =(R.sub.ci1, G.sub.ci1, B.sub.ci1, X.sub.ci1, Y.sub.ci1)(14).

In any cluster C_(i1), a geometrically central pixel Pg_(i1) thereof mayhave its parameter set φ' different from the set of representativeparameter values R_(ci1), G_(ci1), B_(ci1), X_(ci1) and Y_(ci1).

Then, each parameter vector Vp_(xy) has a euclidean relative distanceD_(i1) thereof determined to the representative vector Vp_(i1) of eachcluster C_(i1), such that: ##EQU4##

Each pixel Px(λ(x,y)/α) thus has n relative distances D_(i1) determinedtherefor, including a minimum one D_(min-1), and is labelled with anidentification number i₁ (or an incremented number i₂) of a clusterC_(i1) associated with the minimum distance D_(min-1).

Accordingly, all pixels Px of the picture Pc in the field Fc are eachlabelled with a corresponding one of n identification numbers i₁ (or i₂)(i=1 to n), so that the picture Pc is re-segmented into n second-orderclusters C_(i2) (i=1 to n) each consisting of N_(i2) pixels Px_(i2)labelled with an identical number i₁ (or _(i2)). Also the number N_(i2)of pixels Px_(i2) is variable.

Like operation is repeated, as necessary. In due course, the picture Pcis re-segmented from a set {C_(ij) } of j-th order clusters C_(i1) (i=1to n; j=arbitrary integer) each represented by a representative vectorVp_(ij) in the S[5] field, to a set {C_(i)(j+1) } of j+1-th orderclusters C_(i)(j+1) each represented by a representative vectorVp_(i)(j+1) in the S[5] field.

At j+1=k, if the representative vector Vp_(ik) of the k-th order clusterC_(ik) is equivalent to that Vp_(ij) of the j-th order cluster for eachi, the clustering is converged and hence, when relabelled after the set{C_(ik) }, each pixel Px is kept labelled with a previous cluster numberi_(j-1) (or an incremented number i_(j)). An image segmentation iscompleted with the set {C_(ij) }, so that each cluster of pixels with anidentical number i_(j-1) (or i_(j)) constitutes a final connectedregion.

For the clustering repeated a necessary number of times until aconvergence, the frame memory 210 and the address generator 220 of FIG.5 are adapted to repeat sequentially outputting a synchronized parallelcombination of the color data D210 and the location data D222 of thepicture P210. Further, for an initial setting of blocks C_(i0), theaddress generator 220 is adapted to sequentially output a set of data onaddresses of the central pixels P_(i0) (=Pr_(i0)) to the frame memory210 and the clusternig circuit 230.

FIG. 6 is an exemplarily detailed block diagram of the clusteringcircuit 230 in the conventional system 200 of FIG. 5. This exampleemploys an incremented cluster identification number.

In FIG. 6, designated at reference character 235 is a cluster numbermemory, 236 is a cluster number generator, and 238 is a cluster memory.

The cluster number memory 235 stores therein for each j anidentification number i_(j) (i=1 to n) of each cluster C_(ij), atrespective addresses Ap_(xy) corresponding to coordinates (X,Y) of allof N_(ij) pixels Px(λ/α) as Px_(ij) labelled with the cluster numberi_(j), to thereby update a previous identification number i_(j-1). Inplace of the initial identification number io for any block C_(i0),there is employed a particular integer, such as -1, that will never befound in a value range of i_(j) for 1≦j.

The cluster number generator 236 generates to output for each j asequence of cluster numbers 1_(j) to n_(j), as necessary.

The cluster memory 238 stores therein at least for each j, at eachcluster address Ac_(ij) corresponding to one cluster number i_(j), a setof representative parameter values R_(cij), G_(cij), B_(cij). X_(cij)and Y_(cij) as components of a representative vector Vp_(ij) of anassociated cluster C_(i1), such that:

    R.sub.cij =(ΣR(λ(x,y)/α))/N.sub.ij      (16),

    G.sub.cij =(ΣG(λ(x,y)/α))/N.sub.ij      (17),

    B.sub.cij =(ΣB(λ(x,y)/α))/N.sub.ij      (18),

    X.sub.cij =(ΣX(λ(x,y)/α))/N.sub.ij      (19),

and

    Y.sub.cij =(ΣY(λ(x,y)/α))/N.sub.ij      (20),

where the sum Σ is taken of the N_(ij) pixels Px_(ij). Each addressAc_(ij) is representative of an associated cluster number i_(j), andvice versa.

As shown in FIG. 6, the clustering circuit 230 comprises a distancecalculator 231, a minimum distance estimator 232, a convergenceestimator 234, the cluster number memory 235, the cluster numbergenerator 236, an average calculator 237 and the cluster memory 238.

The distance calculator 231 sequentially calculates, for each pixel Pxeach j, a set {D_(ij) } of euclidean relative distances D_(ij) between aparameter vector Vp_(xy) of the pixel Px(λ(x,y)/α) and respectiverepresentative vectors Vp_(ij) of clusters C_(ij), such that: ##EQU5##

The distance calculation for any pixel Px is made of all the n clustersC_(ij) in an order in which the cluster number i_(j) is output from thegenerator 236, while the order is common to each j. Calculated distancesD_(ij) are output in the same order as the calculation.

The minimum distance estimator 232 functions for each pixel Px each i sothat, upon reception of an m-th distance D_(ij), where "m" is anarbitrary integer such that 1≦m≦n, it has a minimum distance heldtherein as a distance criterion CD_(m-1). from among m-1 distancesD_(ij) it has received till then, and compares the m-th distance D_(ij)with the criterion CD_(m-1) to thereby select from therebetween asmaller one to be held therein as a subsequent criterion CD_(m). Aninitial criterion CD₀ is predetermined to be a maximum permissiblevalue. A final criterion CD_(n) should be a minimum distance D_(min-j)in the set {D_(ij) }, so that an identification number i_(j) of acluster C_(ij) associated therewith should be output as a label for thepixel Px in concern.

The convergence estimator 234 functions for each pixel Px each j tocompare the current cluster number i_(j) output from the estimator 232for the pixel Px in concern with a previous cluster number i_(j-1)stored at a corresponding location in the memory 235, to thereby checkfor a difference or detect a coincidence therebetween. At j=k, if thecoincidence is detected of all pixels Px, an associated clustering isconverged, permitting the data 230 on a result of the clustering to beoutput.

The average calculator 237 sequentially calculates for each j the set ofrepresentative parameter values R_(cij), G_(cij), B_(cij), X_(cij) andY_(cij) of each cluster C_(ij), in accordance with the expressions (16)to (20).

For each frame Nf in which a single picture Pc is processed, theclustering circuit 230 functions as follows.

In an initial phase of the frame Nf=α, the cluster number memory 235,the cluster memory 238, the minimum distance estimator 232 and theconvergence estimator 234 are initialized, and the cluster numbergenerator 236 sequentially outputs to the cluster memory 238 a set ofdata D236a on addresses Ac_(i0) of the blocks (as initial clusters)C_(i0) in the memory 238.

In a synchronized manner therewith, the color data D210 and the locationdata D222 of representative pixels P_(i0) (=P_(i0)) of the blocks C_(i0)are sequentially input from the frame memory 210 (FIG. 5) and theaddress generator 220 (FIG. 5), respectively, to the cluster memory 238,where they are written at the addresses Ac_(i0) designated by theaddress data D236a, so that each address Ac_(i0) has stored therein aset φ' of data D210 and D222 on parameter values R(λ(P_(i0))/α),G(λ(Pr_(i0))/α), B(λ(Pr_(i0))/α), X(λ(Pr_(i0))/α) and Y(λ(P_(i0))/α)) ascomponents of an associated representative vector Vp_(i0).

Then, the cluster number generator 236 outputs the address data D236a inthe order of the cluster number io (i₀ =i_(j) =1 to n) to the clustermemory 238, where the written data D210 and D222 on parameter values{φ'} are sequentially read from their addresses Ac_(i0) in the order ofthe cluster number i₀, to be output as a sequence of data D238 to thedistance calculator 231.

In a synchronized manner therewith, the calculator 231 receives acombination of data D210 and D222 on parameter values R(λ(x,y)/α),G(λ(x,y)/α), B(λ(x,y)/α), X(λ(x,y)/α) and Y(λ(x,y)/α)) of a pixelPx(λ(x,y)/α).

In the calculator 231, the data 238 are sequentially processed by usingthe combination of data D210 and D222 for a calculation in accordancewith the expression (8), to thereby obtain the n relative distancesD_(i0) to be output as a sequence of data D231 to the minimum distanceestimator 232. In synchronism therewith, a set of data D236b eachrepresentative of one cluster number io is sequentially output to theestimator 232.

The estimator 232 processes the data D231 together with the data D236bto hold therein the minimum distance D_(min-0) in combination with anumber io of a corresponding cluster C_(i0), which cluster number i₀ isoutput as a data D232 to the cluster number memory 235.

In synchronism therewith, the data D222 on the parameter valuesX(λ(x,y)/α) and Y(λ(x,y)/α)) of the pixel Px(λ(x,y)/α) are input to thememory 235, where an initial cluster number (e.g. -1) set at an addressAp_(xy) designated by the input data D222 is read and updated by writingthe cluster number i₀ represented by the data 232.

The read cluster number is output to the convergence estimator 234,where it is compared with the cluster number i₀ represented by the data232. A result R₀ of comparison is stored.

The foregoing process after the initial setting of representativeparameter values of blocks C_(i0) is repeated for each pixel Px.

Then, the cluster number generator 236 sequentially outputs a set ofdata D236c representative of cluster numbers i₀, in the order of thenumbers i₀, to the average calculator 237.

In a synchronized manner therewith, the calculator 237 receives asequence of combinations of the color data D210 and the location dataD222 of respective pixels Px(λ(x,y)/α) in the picture Pc, in a presetscan order thereof. Concurrently therewith, the same data as thelocation data D222 in that sequence are input in the preset scan orderto the cluster number memory 235, where the clusters numbers i₀ storedtherein are sequentially read in the same scan order.

Read cluster numbers i₀ are input in the read order to the averagecalculator 237, where they are each respectively compared with a certaincluster number i₀ represented then by one of the data 236c, to therebydetect a coincidence therebetween. Each time when the coincidence isdetected, a corresponding combination of data D210 and D222representative of a parameter vector Vp_(xy) of a pixel Px is processedto be counted and vector-componentwise cumulated.

When the scan of the picture Pc is over for the cluster number i₀,respective cumulated values are divided by a final count number N_(i1)in accordance with the expressions (9) to (13), to thereby determine therepresentative parameter values R_(ci1), G_(ci1), B_(ci1), X_(ci1) andY_(ci1) of the first-order cluster C_(i1), which are output as a set ofdata D237 to the cluster memory 238.

In synchronism therewith, an address data D236a corresponding to thedata D236c in concern is output from the cluster number generator 236 tothe cluster memory 238, where a set of representative parameter valuesR(λ(P_(i0))/α), G(λ(P_(i0))/α), G(λ(P_(i0))/α), X(λ(P_(i0))/α) andY(λ(P_(i0))/α) of a corresponding block C_(i0) are updated by therepresentative parameter values R_(ci1), G_(ci1), B_(ci1), X_(ci1) andY_(ci1) of the first-order cluster C_(i1), respectively.

Like update operation is repeated for each cluster number i₀ designatedby any of the data D236c, so that in the cluster memory 238 respectiveparameter values of all blocks C_(i0) are updated by correspondingparameter values of first-order clusters C_(i1), i.e. n Vp_(i0) areupdated to n Vp_(i1).

Then, similar operations to the block representative vectors {Vp_(i0) }are repeated for the first-order cluster representative vectors {Vp_(i1)}, so that for each pixel Px the relative distances {D_(i1) } arecalculated in accordance with the expression (15) to determine theminimum distance D_(min-1), whereby for each pixel Px a current clusternumber i₁ is provided and compared with a previous culster number i₀.Unless i₁ =i₀ for each pixel Px, {Vp_(i2) } are calculated in accordancewith the expressions (16) to (20), to thereby update {Vp_(i1) }.

Likewise, unless i_(j) =i_(j-1) for each pixel Px, {Vp_(i)(j-1) } areupdated by {Vp_(ij) }.

If i_(k) =i_(k-1) for each pixel Px, a clustering of the picture Pc iscompleted so the cluster numbers i_(k) are sequentially read from thecluster number memory 235 in synchronism with the location data D222 andoutput as the data D230 of segmented regions RS from the clusteringcircuit 230.

In the motion picture processing system 200, an image segmentation ofeach picture Pc is performed by elementwise clustering a union set Ψ ofcharacter parameter sets φ' each consisting of color data D210 andlocation data D222 of a pixel Px therein. For a stationary picture ofeach frame, therefore, a competent segmentation is expectable with afavorable preciseness.

However, the system 200 employs no information on time-dependentvariations between frames for the clustering, in which thus noconsideration is taken of any motion that an object might exhibitbetween pictures, with disadvantages such that a number of pixels withdifferent of motions might be undesirably connected with each other andthat an image region with a uniform motion might be unnecessarilydivided.

In other words, on the one hand, a number of pixels may be inseparablyconnected with each other, if their locations, colors and/or luminancesare vicinal to each other, even when their motions are quite differentfrom each other, thus resulting in an incompetent representation. Morespecifically, for example, when a picture contains a pair of neighboringobjects resemblant in color but different in motion, if the resemblanceof color is significant enough for the conventional image segmentationto cluster them together, they will be found in a single region and willnot be distributed between a pair of separated regions.

On the other hand, an image region substantially uniform in motion butuneven in color and/or luminance may be unconnectably divided, causingan unnecessary local increase in number of regions, resulting in anundesirably biased segmentation due to a restriction from a total numberof regions. For example, when a picture contains a sufficiently smallset of pixels representing a single object colored with a pair ofdifferent colors and moving without revolution, if the differencebetween those colors is significant enough for the conventional imagesegmentation to cluster a subset of the pixel set with either colorseparately from another subset with the other color, the former subsetwill be found in a region disconnetecd from a region including thelatter subset.

The present invention has been achieved with such problems ofconventional data processing methods and systems in mind.

SUMMARY OF THE INVENTION

It is a first object of the present invention to provide a method and asystem for processing a set of data on a temporal sequence of two orthree dimensional pictures with a time-dependent motion intermittentlypicked up therein, by using measures for a compensation of the motion incombination with an image segmentation of the pictures, so that acurrent picture is segmented into a number of regions as clusters ofpixels, and a relative motion is estimated between a respective one ofthe segmented regions in the current picture and a corresponding regionof a previous picture to thereby determine a motion vector.

To achieve the first object, a genus of the present invention provides amotion picture processing method for processing a sequence of motionpictures. The method comprises three steps. In a first step, a currentpicture in the sequence of motion pictures is image-segmented into anumber of first regions. In a second step, for a respective one of thefirst regions, a corresponding second region is determined in a previouspicture to the current picture in the sequence of motion pictures. In athird step, a relative motion is estimated between the respective firstregion and the corresponding second region to determine a motion vectorrepresentative of the relative motion.

According to a species of the genus of the invention, the motion pictureprocessing method further comprises the steps of motion-compensating alocal decoded picture of the previous picture in dependence on themotion vector to obtain a predicted picture of the current picture,subtracting the predicted picture from the current picture to obtain aprediction error picture, coding the prediction error picture in acompressing manner into a set of coded data, decoding the set of codeddata in a decompressing manner into a local decoded error picture, andadding the local decoded error picture to the predicted picture toobtain a local decoded picture of the current picture.

According to another species of the genus of the invention, the currentpicture comprises an arbitrary picture in the sequence of motionpictures, and the previous picture comprises a local decoded picture ofa picture previous to the arbitrary picture in the sequence of motionpictures.

To achieve the first object, another genus of the present inventionprovides a motion picture coding-decoding method for processing asequence of motion pictures to obtain a sequence of decoded pictures.The method comprises thirteen steps. A first step image-segments acurrent picture in the sequence of motion pictures into a number offirst regions. A second step determines for a respective one of thefirst regions a corresponding second region in a previous picture to thecurrent picture in the sequence of motion pictures. A third stepestimates a relative motion between the respective first region and thecorresponding second region to determine a motion vector representativeof the relative motion. A fourth step motion-compensates a local decodedpicture of the previous picture in dependence on the motion vector toobtain a predicted picture of the current picture. A fifth stepsubtracts the predicted picture from the current picture to obtain aprediction error picture. A sixth step encodes the motion vector into asequence of first codes. A seventh step encodes the prediction errorpicture in a compressing manner into a sequence of second codes. Aneighth step multiplexes the sequence of first codes and the sequence ofsecond codes to obtain a multiplexed signal. A ninth step demultiplexesthe multiplexed signal into a combination of a sequence of third codescorresponding to the sequence of first codes and a seqeunce of fourthcodes corresponding to the sequence of second codes. A tenth stepdecodes the sequence of third codes to obtain a decoded vectorcorresponding to the motion vector. An eleventh step decodes thesequence of fourth codes in a decompressing manner to obtain a decodederror picture. A twelfth step responds to the decoded vector tomotion-compensate a previous decoded picture in the sequence of decodedpictures to obtain a decoded prediction picture. A thirteenth step addsthe decoded error picture to the decoded prediction picture to obtain apicture subsequent to the previous decoded picture in the sequence ofdecoded pictures.

To achieve the first object, another genus of the present inventionprovides a picture data processing method for processing a set of dataon a temporal sequence of two or three dimensional pictures with atime-dependent motion intermittently picked up therein, the set of dataincluding a set of current data on a current picture in the sequence ofpictures and a set of previous data on a previous picture to the currentpicture in the sequence of pictures, the current picture consisting of aset of current pixels each respectively defined by a combination ofcurrent image data associated therewith and a combination of currentaddress data representing an address thereof in the current picture, theprevious picture consisting of a set of previous pixels eachrespectively defined by a combination of previous image data associatedtherewith and a combination of previous address data representing anaddress thereof in the previous picture. The method comprises fivesteps. A first step has the set of current data processed so that thecurrent picture is linearly mapped into a spatiotemporal field, as afirst picture consisting of a set of first pixels each respectivelyrepresented by a first character vector equivalent to a sum of a firstimage parameter vector representative of the combination of currentimage data and a first location parameter vector representative of thecombination of current address data. A second step has the set ofprevious data processed so that the previous picture is linearly mappedinto the spatiotemporal field, as a second picture consisting of a setof second pixels each respectively represented by a second charactervector equivalent to a sum of a second image parameter vectorrepresentative of the combination of previous image data and a secondlocation parameter vector representative of the combination of previousaddress data. A third step segments the first picture into a number offirst regions each respectively composed of a cluster of a variablenumber of elements (i.e. first pixels) of the set of first pixels, sothat the first character vectors of the variable number of elements arerelatively vicinal to each other in terms of a distance defined in thespatiotemporal field and have the first location parameter vectorsthereof averaged to obtain a first location representative vectorrepresentative of a location of a correspondent one of the first regionsin the spatiotemporal field. A fourth step determines for a respectiveone of the first regions a corresponding second region in the secondpicture, the corresponding second region being composed of an identicalnumber of elements (i.e. second pixels) of the set of second pixels tothe variable number of elements of the set of first pixels, so that thesecond character vectors of the identical number of elements have thesecond image parameter vectors thereof each respectively relativelyvicinal to a representative vector of the first image parameter vectorsof the first character vectors in terms of the distance and the secondlocation parameter vectors thereof averaged to obtain a second locationrepresentative vector representative of a location of a correspondentone of the second regions in the spatiotemporal field. A fifth stepestimates a relative motion as part of the time-dependent motion betweenthe respective first region and the corresponding second region, bysubtracting the second location representative vector of the latter fromthe first location representative vector of the former to determine amotion vector representive of the relative motion in the spatiotemporalfield.

Moreover, to achieve the first object, another genus of the presentinvention provides a motion picture processing system for processing asequence of motion pictures. The system comprises an image segmentor anda motion estimator. The image segmentor image-segments a current picturein the sequence of motion pictures into a number of first regions. Themotion estimator determines for a respective one of the first regions acorresponding second region in a previous picture to the current picturein the sequence of motion pictures, and estimates a relative motionbetween the respective first region and the corresponding second regionto determine a motion vector representative of the relative motion.

Acoording to a species of this genus of the invention, the motionpicture processing system further comprises a motion compensator formotion-compensating a local decoded picture of the previous picture independence on the motion vector to obtain a predicted picture of thecurrent picture, a subtractor for subtracting the predicted picture fromthe current picture to obtain a prediction error picture, a compressingcoder for coding the prediction error picture in a compressing mannerinto a set of coded data, a decompressing decoder for decoding the setof coded data in a decompressing manner into a local decoded errorpicture, and an adder for adding the restored error picture to thepredicted picture to obtain a local decoded picture of the currentpicture.

According to another species of this genus of the invention, the currentpicture comprises an arbitrary picture in the sequence of motionpictures, and the previous picture comprises a local decoded picture ofa picture previous to the arbitrary picture in the sequence of motionpictures.

To achieve the first object, another genus of the present inventionprovides a motion picture coding-decoding system for processing asequence of motion pictures to obtain a sequence of decoded pictures.The system comprises an image segmentor for image-segmenting a currentpicture in the sequence of motion pictures into a number of firstregions, a motion estimator for determining for a respective one of thefirst regions a corresponding second region in a previous picture to thecurrent picture in the sequence of motion pictures and estimating arelative motion between the respective first region and thecorresponding second region to determine a motion vector representativeof the relative motion, a first motion compensator formotion-compensating a local decoded picture of the previous picture independence on the motion vector to obtain a predicted picture of thecurrent picture, a subtractor for subtracting the predicted picture fromthe current picture to obtain a prediction error picture, a multiplexingencoder for coding the motion vector into a sequence of first codes, forcoding the prediction error picture in a compressing manner into asequence of second codes, and for multiplexing the sequence of firstcodes and the sequence of second codes to obtain a multiplexed signal, ademultiplexing decoder for demultiplexing the multiplexed signal into acombination of a sequence of third codes corresponding to the sequenceof first codes and a seqeunce of fourth codes corresponding to thesequence of second codes, for decoding the sequence of third codes toobtain a decoded vector corresponding to the motion vector, and fordecoding the sequence of fourth codes in a decompressing manner toobtain a decoded error picture, a second motion compensator forresponding to the decoded vector to motion-compensate a previous decodedpicture in the sequence of decoded pictures to obtain a decodedprediction picture, and an adder for adding the decoded error picture tothe decoded prediction picture to obtain a picture subsequent to theprevious decoded picture in the sequence of decoded pictures.

To achieve the first object, another genus of the present inventionprovides a picture data processing system for processing a set of dataon a temporal sequence of two or three dimensional pictures with atime-dependent motion intermittently picked up therein, the set of dataincluding a set of current data on a current picture in the sequence ofpictures and a set of previous data on a previous picture to the currentpicture in the sequence of pictures, the current picture consisting of aset of current pixels each respectively defined by a combination ofcurrent image data associated therewith and a combination of currentaddress data representing an address thereof in the current picture, theprevious picture consisting of a set of previous pixels eachrespectively defined by a combination of previous image data associatedtherewith and a combination of previous address data representing anaddress thereof in the previous picture. The system comprises a set ofdata processing measures such as a set of programs in a controller, animage segmentor and a motion estimator. The data processing measurescooperate with each other to have the set of current data processed sothat the current picture is linearly mapped into a spatiotemporal field,as a first picture consisting of a set of first pixels each respectivelyrepresented by a first character vector equivalent to a sum of a firstimage parameter vector representative of the combination of currentimage data and a first location parameter vector representative of thecombination of current address data, and to have the set of previousdata processed so that the previous picture is linearly mapped into thespatiotemporal field, as a second picture consisting of a set of secondpixels each respectively represented by a second character vectorequivalent to a sum of a second image parameter vector representative ofthe combination of previous image data and a second location parametervector representative of the combination of previous address data. Theimage segmentor segments the first picture into a number of firstregions each respectively composed of a cluster of a variable number ofelements of the set of first pixels, so that the first character vectorsof the variable number of elements are relatively vicinal to each otherin terms of a distance defined in the spatiotemporal field and have thefirst location parameter vectors thereof averaged to obtain a firstlocation representative vector representative of a location of acorrespondent one of the first regions in the spatiotemporal field. Themotion estimator determines for a respective one of the first regions acorresponding second region in the second picture, the correspondingsecond region being composed of an identical number of elements of theset of second pixels to the variable number of elements of the set offirst pixels, so that the second character vectors of the identicalnumber of elements have the second image parameter vectors thereof eachrespectively relatively vicinal to a representative vector of the firstimage parameter vectors of the first character vectors in terms of thedistance and the second location parameter vectors thereof averaged toobtain a second location representative vector representative of alocation of a correspondent one of the second regions in thespatiotemporal field, and estimates a relative motion as part of thetime-dependent motion between the respective first region and thecorresponding second region, by subtracting the second locationrepresentative vector of the latter from the first locationrepresentative vector of the former to determine a motion vectorrepresentive of the relative motion in the spatiotemporal field.

Therefore, according to any genus of the invention for the first object,a motion vector represents a relative motion estimated between a clusterof pixels in a current picture, which pixels have their characterparameter sets componentwise vicinal to each other, and a correspondingregion of a previous picture, so that a significant portion of a movingobject picked up in the cluster may be picked up in the correspondingregion, with a higher probability than the case of equi-divided blocks,and hence with an increased tendency to faithfully or naturallyrepresent a picked-up motion, thus resulting in an improved motioncompensation.

It is a second object of the present invention to provide a method and asystem for processing a set of data on a temporal sequence of two orthree dimensional pictures with a time-dependent motion intermittentlypicked up therein, by using measures for a compensation of the motion incombination with an image segmentation of the pictures, so that arelative motion is estimated between a respective one of minute-dividedpieces of a current picture and a corresponding minute piece in aprevious picture to thereby determine a parameter respresentative of therelative motion, while any piece may comprise one or more pixels, andthe current picture is segmented into a number of regions consisting ofpixels clustered with respect to a set of character parameters includingthe motion-representative parameter.

To achieve the second object, a genus of the present invention providesa motion picture processing method for processing a sequence of motionpictures including a current picture and a previous picture thereto. Thecurrent picture consists of a set of current pixels. The previouspicture consists of a set of previous pixels. The current and previouspixels are each respectively defined by a combination of image dataassociated therewith and a combination of address data thereof. Themotion picture processing method comprises five or six steps. A firststep divides the current picture into a predetermined number of currentminute pieces each respectively consisting of one or more elements (i.e.pixels) of the set of current pixels. A second step may divide theprevious picture into a number of previous minute pieces identical tothe predetermined number. The previous minute pieces may eachrespectively consist of one or more elements (i.e. pixels) of the set ofprevious pixels. A third step determines, for a respective one of thepredetermined number of current minute pieces, a corresponding previousminute piece. A fourth step estimates a relative motion between therespective current minute piece and the corresponding previous minutepiece. A fifth step additionally defines each element of the set ofcurrent pixels in the respective current minute piece, by a combinationof motion data representative of the relative motion. A sixth stepserves for image-segmenting the current picture into a number of firstregions with respect to the combination of image data, the combinationof address data and the combination of motion data. The first regionsare smaller in number than the current minute pieces.

According to a species of this genus of the invention, the motionpicture processing method further comprises the steps ofimage-mosaicking the current picture to obtain a predicted picturethereof so that respective elements of a subset of the set of currentpixels associated with a respective one of the first regions have acombination of region-representative image data in place of therespective combinations of image data thereof, subtracting the predictedpicture from the current picture to obtain a prediction error picture,coding the predicted picture and the prediction error in a compressingmanner into a first set of coded data and a second set of coded data,respectively, decoding the first set of coded data and the second set ofcoded data in a decompressing manner into a local decoded predictionpicture and a local decoded error picture, respectively, and adding thelocal decoded error picture to the local decoded prediction picture toobtain a local decoded picture.

According to another species of this genus of the invention, the currentpicture comprises an arbitrary picture in the sequence of motionpictures, and the previous picture comprises a local decoded picture ofa picture previous to that arbitrary picture in the sequence of motionpictures.

To achieve the second object, another genus of the present inventionprovides a motion picture coding-decoding method for processing asequence of motion pictures including a current picture and a previouspicture thereto. The current picture consists of a set of currentpixels. The previous picture consists of a set of previous pixels. Thecurrent and previous pixels are each respectively defined by acombination of image data associated therewith and a combination ofaddress data thereof. The coding-decoding method comprises fourteen orfifteen steps. A first step divides the current picture into apredetermined number of current minute pieces each respectivelyconsisting of one or more elements of the set of current pixels. Asecond step may divide the previous picture into an identical number ofprevious minute pieces to the predetermined number, which previousminute pieces may each respectively consist of one or more elements ofthe set of previous pixels. A third step determines for a respective oneof the current minute pieces a corresponding previous minute piece. Afourth step estimates a relative motion between the respective currentminute piece and the corresponding previous minute piece. A fifth stepadditionally defines each element of the set of current pixels in therespective current minute piece, by a combination of motion datarepresentative of the relative motion. A sixth step serves forimage-segmenting the current picture into a number of first regions withrespect to the combination of image data, the combination of addressdata and the combination of motion data, while the first regions aresmaller in number than the current minute pieces. A seventh step servesfor image-mosaicking the current picture to obtain a predicted picturethereof so that respective elements of a subset of the set of currentpixels associated with a respective one of the first regions have acombination of region-representative image data in place of thecombinations of image data thereof. An eighth step subtracts thepredicted picture from the current picture to obtain a prediction errorpicture. A ninth step encodes the predicted picture in a compressingmanner into a sequence of first codes. A tenth step encodes theprediction error picture in a compressing manner into a sequence ofsecond codes. An eleventh step multiplexes the sequence of first codesand the sequence of second codes to obtain a multiplexed signal. Atwelfth step demultiplexes the multiplexed signal into a combination ofa sequence of third codes corresponding to the sequence of first codesand a sequence of fourth codes corresponding to the sequence of secondcodes. A thirteenth step decodes the sequence of third codes in adecompressing manner into a decoded prediction picture. A fourteenthstep decodes the sequence of fourth codes in a decompressing manner intoa decoded error picture. A fifteenth step adds the decoded error pictureto the decoded prediction picture to obtain a decoded picturecorresponding to the current picture.

To achieve the second object, another genus of the present inventionprovides a picture data processing method for processing a set of dataon a temporal sequence of two or three dimensional pictures with atime-dependent motion intermittently picked up therein. The set of dataincludes a set of current data on a current picture in the sequence ofpictures and a set of previous data on a previous picture to the currentpicture therein. The current picture consists of a set of current pixelseach respectively defined by a combination of current image dataassociated therewith and a combination of current address datarepresenting an address thereof in the current picture. The previouspicture consists of a set of previous pixels each respectively definedby a combination of previous image data associated therewith and acombination of previous address data representing an address thereof inthe previous picture. The picture data processing method comprises sixor seven steps. A first step has the set of current data processed sothat the current picture is linearly mapped into a spatiotemporal field,as a first picture consisting of a set of first pixels each respectivelyrepresented by a first character vector equivalent to a sum of a firstimage parameter vector representative of the combination of currentimage data and a first location parameter vector representative of thecombination of current address data. A second step has the set ofprevious data processed so that the previous picture is linearly mappedinto the spatiotemporal field, as a second picture consisting of a setof second pixels each respectively represented by a second charactervector equivalent to a sum of a second image parameter vectorrepresentative of the combination of previous image data and a secondlocation parameter vector representative of the combination of previousaddress data. A third step divides the first picture into apredetermined number of first minute pieces each respectively consistingof one or more elements (i.e. pixels) of the set of first pixels havingthe first location parameter vectors thereof averaged to obtain a firstlocation representative vector representative of a location of acorrespondent one of the first minute pieces in the spatiotemporalfield. A fourth step may divide the second picture into an identicalnumber of second minute pieces to the predetermined number. The secondminute pieces may each respectively consist of one or more elements(i.e. pixels) of the set of second pixels having the second locationparameter vectors thereof averaged to obtain a second locationrepresentative vector representative of a location of a correspondingone of the second minute pieces in the spatiotemporal field. A fifthstep determines, for a respective one of the first minute pieces, acorresponding second minute piece. A sixth step estimates a relativemotion as part of the time-dependent motion between the respective oneof the first minute pieces and the corresponding second minute piece, bysubtracting a second location representative vector of the latter fromthe first location representative vector of the former to determine amotion vector representive of the relative motion in the spatiotemporalfield. A seventh step adds the motion vector to the first charactervector of each element of the set of first pixels in the respective oneof the first minute pieces to obtain a dimension-increased charactervector. An eighth step segments the first picture into a number of firstregions each respectively composed of a cluster of a variable number ofelements of the set of first pixels, so that the dimension-increasedcharacter vectors of the variable number of elements are relativelyvicinal to each other in terms of a distance defined in thespatiotemporal field.

Moreover, to achieve the second object, another genus of the presentinvention provides a motion picture processing system for processing asequence of motion pictures including a current picture and a previouspicture thereto. The current picture consists of a set of currentpixels. The previous picture consists of a set of previous pixels. Thecurrent and previous pixels are each respectively defined by acombination of image data associated therewith and a combination ofaddress data thereof. The motion picture processing system comprises amotion estimator and an image segmentor. The motion estimator dividesthe current picture into a predetermined number of current minute pieceseach respectively consisting of one or more elements (i.e. pixels) ofthe set of current pixels, and may divide the previous picture into anidentical number of previous minute pieces to the predetermined numberof current minute pieces. The previous minute pieces may eachrespectively consist of one or more elements (i.e. pixels) of the set ofprevious pixels. The motion estimator then determines for a respectiveone of the current minute pieces a corresponding previous minute piece,estimates a relative motion between the respective one of the currentminute pieces and the corresponding previous minute piece, andadditionally defines each element of the set of current pixels in therespective one of the current minute pieces, by a combination of motiondata representative of the relative motion. The image segmentor servesfor image-segmenting the current picture into a number of first regionswith respect to the combination of image data, the combination ofaddress data and the combination of motion data, while the number offirst regions is smaller than the predetermined number of current minutepieces.

According to a species of this genus of the invention, the imagesegmentor further serves for image-mosaicking the current picture toobtain a predicted picture thereof so that respective elements (i.e.pixels) of a subset (i.e. in a region) of the set of current pixelsassociated with a respective one of the first regions have a combinationof region-representative image data in place of the combinations ofimage data thereof, and the motion picture processing system furthercomprises a subtractor for subtracting the predicted picture from thecurrent picture to obtain a prediction error picture, a compressingcoder for coding the predicted picture and the prediction error picturein a compressing manner into a first set of coded data and a second setof coded data, respectively, a decompressing decoder for decoding thefirst set of coded data and the second set of coded data in adecompressing manner into a local decoded prediction picture and a localdecoded error picture, respectively, and an adder for adding the localdecoded error picture to the local decoded prediction picture to obtaina local decoded picture.

According to another species of this genus of the invention, the currentpicture comprises an arbitrary picture in the sequence of motionpictures, and the previous picture comprises a local decoded picture ofa picture previous to the arbitrary picture in the sequence of motionpictures.

To achieve the second object, another genus of the present inventionprovides a motion picture coding-decoding system for processing asequence of motion pictures including a current picture and a previouspicture thereto. The current picture consists of a set of currentpixels. The previous picture consists of a set of previous pixels. Thecurrent and previous pixels are each respectively defined by acombination of image data associated therewith and a combination ofaddress data thereof. The motion picture coding-decoding systemcomprises a motion estimator, an image segmentor, a subtractor, amultiplexing encoder, a demultiplexing decoder and an adder. The motionestimator divides the current picture into a predetermined number ofcurrent minute pieces each respectively consisting of one or moreelements of the set of current pixels, and may divide the previouspicture into an identical number of previous minute pieces to thepredetermined number, the previous minute pieces each respectivelyconsisting of one or more elements of the set of previous pixels,determines for a respective one of the current minute pieces acorresponding previous minute piece, estimates a relative motion betweenthe respective one of the current minute pieces and the correspondingprevious minute piece, and additionally defines each element of the setof current pixels in the respective one of the current minute pieces, bya combination of motion data representative of the relative motion. Theimage segmentor serves for image-segmenting the current picture into anumber of first regions with respect to the combination of image data,the combination of address data and the combination of motion data,while the number of first regions is smaller than the predeterminednumber, and for image-mosaicking the current picture to obtain apredicted picture thereof so that respective elements of a subset of theset of current pixels associated with a respective one of the firstregions have a combination of region-representative image data in placeof the combinations of image data thereof. The subtractor subtracts thepredicted picture from the current picture to obtain a prediction errorpicture. The multiplexing encoder codes the predicted picture and theprediction error picture in a compressing manner into a sequence offirst codes and a sequence of second codes, respectively, andmultiplexes the sequence of first codes and the sequence of second codesto obtain a multiplexed signal. The demultiplexing decoder demultiplexesthe multiplexed signal into a combination of a sequence of third codescorresponding to the sequence of first codes and a seqeunce of fourthcodes corresponding to the sequence of second codes, and decodes thesequence of third codes and the sequence of fourth codes in adecompressing manner into a decoded prediction picture and a decodederror picture, respectively. The adder adds the decoded error picture tothe decoded prediction picture to obtain a decoded picture correspondingto the current picture.

To achieve the second object, another genus of the present inventionprovides a picture data processing system for processing a set of dataon a temporal sequence of two or three dimensional pictures with atime-dependent motion intermittently picked up therein. The set of dataincludes a set of current data on a current picture in the sequence ofpictures and a set of previous data on a previous picture to the currentpicture in the sequence of pictures. The current picture consists of aset of current pixels each respectively defined by a combination ofcurrent image data associated therewith and a combination of currentaddress data representing an address thereof in the current picture. Theprevious picture consists of a set of previous pixels each respectivelydefined by a combination of previous image data associated therewith anda combination of previous address data representing an address thereofin the previous picture.

The picture data processing system comprises a set of data processingmeasures such as a set of programs, a motion estimator and an imagesegmentor. The data processing measures serve for having the set ofcurrent data processed so that the current picture is linearly mappedinto a spatiotemporal field, as a first picture consisting of a set offirst pixels each respectively represented by a first character vectorequivalent to a sum of a first image parameter vector representative ofthe combination of current image data and a first location parametervector representative of the combination of current address data, andfor having the set of previous data processed so that the previouspicture is linearly mapped into the spatiotemporal field, as a secondpicture consisting of a set of second pixels each respectivelyrepresented by a second character vector equivalent to a sum of a secondimage parameter vector representative of the combination of previousimage data and a second location parameter vector representative of thecombination of previous address data. The motion estimator divides thefirst picture into a predetermined number of first minute pieces eachrespectively consisting of one or more elements of the set of firstpixels having the first location parameter vectors thereof averaged toobtain a first location representative vector representative of alocation of a correspondent one of the first minute pieces in thespatiotemporal field, may divide the second picture into an identicalnumber of second minute pieces to the predetermined number, the secondminute pieces each respectively consisting of one or more elements ofthe set of second pixels having the second location parameter vectorsthereof averaged to obtain a second location representative vectorrepresentative of a location of a correspondent one of the second minutepieces in the spatiotemporal field, determines for a respective one ofthe first minute pieces a corresponding second minute piece, estimates arelative motion as part of the time-dependent motion between therespective one of the first minute pieces and the corresponding secondminute piece, by subtracting a second location representative vector ofthe latter from the first location representative vector of the formerto determine a motion vector representive of the relative motion in thespatiotemporal field, and adds the motion vector to the first charactervector of each element of the set of first pixels in the respective oneof the first minute pieces to obtain a dimension-increased charactervector. The image segmentor is implemented for segmenting the firstpicture into a number of first regions each respectively composed of acluster of a variable number of elements of the set of first pixels, sothat the dimension-increased character vectors of the variable number ofelements are relatively vicinal to each other in terms of a distancedefined in the spatiotemporal field.

Still more, to achieve the second object, another genus of the inventionprovides a motion picture processing method for processing a motionpicture by using a reference picture to obtain an image-segmentedpicture of the motion picture. The motion picture processing methodcomprises six steps. A first step stores a set of pixel datarepresentative of the motion picture in an accessible manner as acurrent frame picture having a set of image data of a set of pixelsthereof. A second step stores a set of pixel data representative of thereference picture in an accessible manner as a reference frame picture.A third step estimates a set of motion vectors between the current framepicture and the reference frame picture. A fourth step stores a set ofmotion data representative of the set of motion vectors in an accessiblemanner. A fifth step generates a series of address signals for thecurrent frame picture to obtain a set of location data of the set ofpixels. A sixth step serves for image-segmenting the current framepicture by clustering the set of pixels with respect to the set of imagedata, the set of motion data and the set of location data.

According to a species of this genus of the invention, the set of pixeldata representative of the motion picture comprises the set of imagedata, and the set of image data comprises a set of color image data. Theset of pixel data representative of the reference picture may compriseanother set of color image data.

According to another species of this genus of the invention, theclustering is performed in a weighting manner by a factor having a valuethereof controlled in correspondence to a character parameter of thecurrent frame picture.

According to another species of this genus of the invention, the motionpicture processing method further comprises the step of image-segmentingthe current frame picture to obtain an image-segmented picture includinga variety of minute regions, and a step of eliminating the minuteregions from the image-segmented picture.

According to another species of this genus of the invention, the motionpicture processing method further comprises a step of subjecting anarbitrary one of the set of image data, the set of motion data and theset of location data to a noise filter, before the clustering.

Yet more, to achieve the second object, another genus of the presentinvention provides a motion picture processing system for processing amotion picture by using a reference picture to obtain an image-segmentedpicture of the motion picture. The motion picture processing systemcomprises a first memory, a second memory, a motion estimator, a thirdmemory, an address generator and an image segmentor. The first memorystores therein a set of pixel data representative of the motion picturein an accessible manner as a current frame picture having a set of imagedata of a set of pixels thereof. The second memory stores therein a setof pixel data representative of the reference picture in an accessiblemanner as a reference frame picture. The motion estimator estimates aset of motion vectors between the current frame picture and thereference frame picture. The third memory stores therein a set of motiondata representative of the set of motion vectors in an accessiblemanner. The address generator generates a series of address signals forthe current frame picture to obtain a set of location data of the set ofpixels. The image segmentor serves for image-segmenting the currentframe picture by clustering the set of pixels with respect to the set ofimage data, the set of motion data and the set of location data.

According to a species of this genus of the invention, the set of pixeldata representative of the motion picture comprises the set of imagedata, and the set of image data comprises a set of color image data. Theset of pixel data representative of the reference picture may compriseanother set of color image data.

According to another species of this genus of the invention, theclustering is performed in a weighting manner by a factor having a valuethereof controlled in correspondence to a character parameter of thecurrent frame picture.

According to another species of this genus of the invention, the motionpicture processing system further comprises the image segmentor forimage-segmenting the current frame picture to obtain an image-segmentedpicture including a variety of minute regions, and an eliminator foreliminating the minute regions from the image-segmented picture.

According to another species of this genus of the invention, the motionpicture processing system further comprises a noise filter for filteringan arbitrary one of the set of image data, the set of motion data andthe set of location data.

Therefore, according to any genus of the invention for the secondobject, a flexible clustering of pixels is performed with respect to aset of character parameters including a representative one of a relativemotion estimated between a minute piece of a current picture and acorresponding piece of a previous picture, so that in the currentpicture a pair of pieces different of motion may either have its pixelsclustered in a region and the other have its pixels clustered in anotherregion, or a pair of pieces alike in motion may have their pixelsclustered all in a single region, with an increased tendency tofaithfully or naturally represent an original region connected in apicked-up image, thus resulting in an improved image segmentation.

For example, in a current picture, some piece may represent a certainportion of a moving object, and another, a neighboring portion thereof.An estimated motion of the former should however be identical to that ofthe latter, thus causing associated pixels to be clustered in aconnected region free of an erroneous contour or discontinuity,resulting in a high-grade picture quality.

It is a third object of the present invention to provide a method and asystem for processing a set of data on a temporal sequence of two orthree dimensional pictures with a time-dependent motion intermittentlypicked up therein, by using measures for a compensation of the motion incombination with an image segmentation of the pictures, in a feedbackingmanner so that a current picture is segmented into a number of regionsas clusters of pixels, a relative motion is estimated between arespective one of sub-regions of a respective one of the segmentedregions in the current picture and a corresponding sub-region in aprevious picture to thereby determine a representative parameter of therelative motion, while any sub-region may comprise one or more pixels,and the current picture is resegmented into a number of regionsconsisting of pixels reclustered with respect to a set of characterparameters including the motion-representative parameter.

To achieve the third object, a genus of the present invention provides amotion picture processing method for processing a sequence of motionpictures including a current picture and a previous picture thereto. Thecurrent picture consists of a set of current pixels. The previouspicture consists of a set of previous pixels. The current and previouspixels are each respectively defined by a combination of image dataassociated therewith and a combination of address data thereof. Themotion picture processing method comprises six steps. A first stepimage-segments the current picture into a predetermined number ofcurrent regions. A second step divides a respective one of the currentregions into a variable number of sub-regions each respectivelyconsisting of one or more elements of the set of current pixels. A thirdstep determines for a respective one of the sub-regions of therespective one of the current regions a corresponding small region inthe previous picture. A fourth step estimates a relative motion betweenthe respective one of the sub-regions of the respective one of thecurrent regions and the corresponding small region. A fifth stepadditionally defines each element of the set of current pixels in therespective one of the sub-regions of the respective one of the currentregions, by a combination of motion data representative of the relativemotion. A sixth step resegments the current picture into a number offirst regions with respect to the combination of image data, thecombination of address data and the combination of motion data.

According to a species of this genus of the invention, the motionpicture processing method further comprises a step of image-mosaickingthe current picture to obtain a predicted picture thereof so thatrespective elements of a subset of the set of current pixels associatedwith a respective one of the first regions have a combination ofregion-representative image data in place of the combinations of imagedata thereof, a step of subtracting the predicted picture from thecurrent picture to obtain a prediction error picture, a step of codingthe predicted picture in a compressing manner into a first set of codeddata, a step of coding the prediction error picture in a compressingmanner into a second set of coded data, a step of decoding the first setof coded data in a decompressing manner into a local decoded predictionpicture, a step of decoding the second set of coded data in adecompressing manner into a local decoded error picture, and a step ofadding the local decoded error picture to the local decoded predictionpicture to obtain a local decoded picture.

According to another species of this genus of the invention, the currentpicture comprises an arbitrary picture in the sequence of motionpictures, and the previous picture comprises a local decoded picture ofa picture previous to the arbitrary picture in the sequence of motionpictures.

To achieve the third object, another genus of the present inventionprovides a motion picture coding-decoding method for processing asequence of motion pictures including a current picture and a previouspicture thereto. The current picture consists of a set of currentpixels. The previous picture consists of a set of previous pixels. Thecurrent and previous pixels are each respectively defined by acombination of image data associated therewith and a combination ofaddress data thereof. The motion picture coding-decoding methodcomprises fifteen steps. A first step is for image-segmenting thecurrent picture into a predetermined number of current regions. A secondstep divides a respective one of the current regions into a variablenumber of sub-regions each respectively consisting of one or moreelements of the set of current pixels. A third step determines for arespective one of the sub-regions of the respective one of the currentregions a corresponding small region in the previous picture. A fourthstep estimates a relative motion between the respective one of thesub-regions of the respective one of the current regions and thecorresponding small region. A fifth step additionally defines eachelement of the set of current pixels in the respective one of thesub-regions of the respective one of the current regions, by acombination of motion data representative of the relative motion. Asixth step resegments the current picture into a number of first regionswith respect to the combination of image data, the combination ofaddress data and the combination of motion data. A seventh step is forimage-mosaicking the current picture to obtain a predicted picturethereof so that respective elements of a subset of the set of currentpixels associated with a respective one of the first regions have acombination of region-representative image data in place of thecombinations of image data thereof. An eighth step subtracts thepredicted picture from the current picture to obtain a prediction errorpicture. A ninth step codes the predicted picture in a compressingmanner into a sequence of first codes. A tenth step codes the predictionerror picture in a compressing manner into a sequence of second codes.An eleventh step multiplexes the sequence of first codes and thesequence of second codes to obtain a multiplexed signal. A twelfth stepdemultiplexes the multiplexed signal into a combination of a sequence ofthird codes corresponding to the sequence of first codes and a seqeunceof fourth codes corresponding to the sequence of second codes. Athirteenth step decodes the sequence of third codes in a decompressingmanner into a decoded prediction picture. A fourteenth step decodes thesequence of fourth codes in a decompressing manner into a decoded errorpicture. A fifteenth step adds the decoded error picture to the decodedprediction picture to obtain a decoded picture corresponding to thecurrent picture.

To achieve the third object, another genus of the present inventionprovides a picture data processing method for processing a set of dataon a temporal sequence of two or three dimensional pictures with atime-dependent motion intermittently picked up therein. The set of dataincludes a set of current data on a current picture in the sequence ofpictures and a set of previous data on a previous picture to the currentpicture in the sequence of pictures. The current picture consists of aset of current pixels each respectively defined by a combination ofcurrent image data associated therewith and a combination of currentaddress data representing an address thereof in the current picture. Theprevious picture consists of a set of previous pixels each respectivelydefined by a combination of previous image data associated therewith anda combination of previous address data representing an address thereofin the previous picture. The picture data processing method compriseseight steps. A first step has the set of current data processed so thatthe current picture is linearly mapped into a spatiotemporal field, as afirst picture consisting of a set of first pixels each respectivelyrepresented by a first character vector equivalent to a sum of a firstimage parameter vector representative of the combination of currentimage data and a first location parameter vector representative of thecombination of current address data. A second step has the set ofprevious data processed so that the previous picture is linearly mappedinto the spatiotemporal field, as a second picture consisting of a setof second pixels each respectively represented by a second charactervector equivalent to a sum of a second image parameter vectorrepresentative of the combination of previous image data and a secondlocation parameter vector representative of the combination of previousaddress data. A third step segments the first picture into apredetermined number of segment regions each respectively composed of afirst cluster of a variable number of elements of the set of firstpixels, so that the first character vectors of the variable number ofelements are relatively vicinal to each other in terms of a distancedefined in the spatiotemporal field. A fourth step divides a respectiveone of the segment regions into a variable number of sub-regions eachrespectively consisting of one or more elements of the set of firstpixels having the first location parameter vectors thereof averaged toobtain a first location representative vector representative of alocation of a correspondent one of the sub-regions in the spatiotemporalfield. A fifth step determines for a respective one of the sub-regionsof the respective one of the segment regions a corresponding smallregion in the second picture, which small region is composed of anidentical number of elements of the set of second pixels to the one ormore elements of the set of first pixels in the respective one of thesub-regions, so that the second character vectors of the identicalnumber of elements have the second image parameter vectors thereof eachrespectively relatively vicinal to a representative vector of the firstimage parameter vectors of the first character vectors in saidrespective one of said variable number of sub-regions in terms of thedistance and the second location parameter vectors thereof averaged toobtain a second location representative vector representative of alocation of the corresponding small region in said spatiotemporal field.A sixth step estimates a relative motion as part of the time-dependentmotion between the respective one of the sub-regions and thecorresponding small region, by subtracting the second locationrepresentative vector of the corresponding small region from the firstlocation representative vector of the respective one of the sub-regionsto determine a motion vector representive of the relative motion in thespatiotemporal field. A seventh step adds the motion vector to the firstcharacter vector of each element of the set of first pixels in therespective one of the sub-regions of the respective one of the segmentregions to obtain a dimension-increased character vector. An eighth stepresegments the first picture into a number of first regions eachrespectively composed of a second cluster of a variable number ofelements of the set of first pixels, so that the dimension-increasedcharacter vectors of the variable number of elements are relativelyvicinal to each other in terms of the distance in the spatiotemporalfield.

Moreover, to achieve the third object, another genus of the inventionprovides a motion picture processing system for processing a sequence ofmotion pictures including a current picture and a previous picturethereto. The current picture consists of a set of current pixels. Theprevious picture consists of a set of previous pixels. The current andprevious pixels are each respectively defined by a combination of imagedata associated therewith and a combination of address data thereof. Themotion picture processing system comprises a segmentor, a motionestimator and a resegmentor. The segmentor image-segments the currentpicture into a predetermined number of current regions. The motionestimator divides a respective one of the current regions into avariable number of sub-regions each respectively consisting of one ormore elements of the set of current pixels, determines for a respectiveone of the sub-regions of the respective one of the current regions acorresponding small region in the previous picture, estimates a relativemotion between the respective one of the sub-regions of the respectiveone of the current regions and the corresponding small region, andadditionally defines each element of the set of current pixels in therespective one of the sub-regions of the respective one of the currentregions, by a combination of motion data representative of the relativemotion. The resegmentor resegments the current picture into a number offirst regions with respect to the combination of image data, thecombination of address data and the combination of motion data.

According to a species of this genus of the invention, the motionpicture processing system further comprises the resegmentor forimage-mosaicking the current picture to obtain a predicted picturethereof so that respective elements of a subset of the set of currentpixels associated with a respective one of the first regions have acombination of region-representative image data in place of thecombinations of image data thereof, a subtractor for subtracting thepredicted picture from the current picture to obtain a prediction errorpicture, a compressing coder for coding the predicted picture and theprediction error picture in a compressing manner into a first set ofcoded data and a second set of coded data, respectively, a decompressingdecoder for decoding the first set of coded data and the second set ofcoded data in a decompressing manner into a local decoded predictionpicture and a local decoded error picture, respectively, and an adderfor adding the local decoded error picture to the local decodedprediction picture to obtain a local decoded picture.

According to another species of this genus of the invention, the currentpicture comprises an arbitrary picture in the sequence of motionpictures, and the previous picture comprises a local decoded picture ofa picture previous to the arbitrary picture in the sequence of motionpictures.

To achieve the third object, another genus of the present inventionprovides a motion picture coding-decoding system for processing asequence of motion pictures including a current picture and a previouspicture thereto. The current picture consists of a set of currentpixels. The previous picture consists of a set of previous pixels. Thecurrent and previous pixels are each respectively defined by acombination of image data associated therewith and a combination ofaddress data thereof. The motion picture coding-decoding systemcomprises a segmentor, a motion estimator, a resegmentor, a subtractor,a multiplexing encoder, a demultiplexing decoder and an adder. Thesegmentor is for image-segmenting the current picture into apredetermined number of current regions. The motion estimator divides arespective one of the current regions into a variable number ofsub-regions each respectively consisting of one or more elements of theset of current pixels, determines for a respective one of thesub-regions of the respective one of the current regions a correspondingsmall region in the previous picture, estimates a relative motionbetween the respective one of the sub-regions of the respective one ofthe current regions and the corresponding small region, and additionallydefines each element of the set of current pixels in the respective oneof the sub-regions of the respective one of the current regions, by acombination of motion data representative of the relative motion. Theresegmentor resegments the current picture into a number of firstregions with respect to the combination of image data, the combinationof address data and the combination of motion data, and image-mosaicksthe current picture to obtain a predicted picture thereof so thatrespective elements of a subset of the set of current pixels associatedwith a respective one of the first regions have a combination ofregion-representative image data in place of the combinations of imagedata thereof. The subtractor subtracts the predicted picture from thecurrent picture to obtain a prediction error picture. The encoder codesthe predicted picture and the prediction error picture in a compressingmanner into a sequence of first codes and a sequence of second codes,respectively, and multiplexes the sequence of first codes and thesequence of second codes to obtain a multiplexed signal. The decoderdemultiplexes the multiplexed signal into a combination of a sequence ofthird codes corresponding to the sequence of first codes and a seqeunceof fourth codes corresponding to the sequence of second codes, anddecodes the sequence of third codes and the sequence of fourth codes ina decompressing manner into a decoded prediction picture and a decodederror picture, respectively. The adder adds the decoded error picture tothe decoded prediction picture to obtain a decoded picture correspondingto the current picture.

To achieve the third object, another genus of the present inventionprovides a picture data processing system for processing a set of dataon a temporal sequence of two or three dimensional pictures with atime-dependent motion intermittently picked up therein. The set of dataincludes a set of current data on a current picture in the sequence ofpictures and a set of previous data on a previous picture to the currentpicture in the sequence of pictures. The current picture consists of aset of current pixels each respectively defined by a combination ofcurrent image data associated therewith and a combination of currentaddress data representing an address thereof in the current picture. Theprevious picture consists of a set of previous pixels each respectivelydefined by a combination of previous image data associated therewith anda combination of previous address data representing an address thereofin the previous picture. The picture data processing system comprises aset of data processing measures such as a set of programs, a segmentorand a motion estimator. The data processing measures serve for havingthe set of current data processed so that the current picture islinearly mapped into a spatiotemporal field, as a first pictureconsisting of a set of first pixels each respectively represented by afirst character vector equivalent to a sum of a first image parametervector representative of the combination of current image data and afirst location parameter vector representative of the combination ofcurrent address data, and for having the set of previous data processedso that the previous picture is linearly mapped into the spatiotemporalfield, as a second picture consisting of a set of second pixels eachrespectively represented by a second character vector equivalent to asum of a second image parameter vector representative of the combinationof previous image data and a second location parameter vectorrepresentative of the combination of previous address data. Thesegmentor segments the first picture into a predetermined number ofsegment regions each respectively composed of a first cluster of avariable number of elements of the set of first pixels, so that thefirst character vectors of the variable number of elements arerelatively vicinal to each other in terms of a distance defined in thespatiotemporal field. The motion estimator divides a respective one ofthe segment regions into a variable number of sub-regions eachrespectively consisting of one or more elements of the set of firstpixels having the first location parameter vectors thereof averaged toobtain a first location representative vector representative of alocation of a correspondent one of the sub-regions in the spatiotemporalfield, determines for a respective one of the sub-regions of therespective one of the segment regions a corresponding small region inthe second picture, which small region is composed of an identicalnumber of elements of the set of second pixels to the one or moreelements of the set of first pixels in the respective one of thesub-regions, so that the second character vectors of the identicalnumber of elements have the second image parameter vectors thereof eachrespectively relatively vicinal to a representative vector of the firstimage parameter vectors of the first character vectors in saidrespective one of said variable number of sub-regions in terms of thedistance and the second location parameter vectors thereof averaged toobtain a second location representative vector representative of alocation of the corresponding small region in said spatiotemporal field,estimates a relative motion as part of the time-dependent motion betweenthe respective one of the sub-regions and the corresponding smallregion, by subtracting the second location representative vector of thecorresponding small region from the first location representative vectorof the respective one of the sub-regions to determine a motion vectorrepresentive of the relative motion in the spatiotemporal field, andadds the motion vector to the first character vector of each element ofthe set of first pixels in the respective one of the sub-regions of therespective one of the segment regions to obtain a dimension-increasedcharacter vector. The segmentor is implemented to resegment the firstpicture into a number of first regions each respectively composed of asecond cluster of a variable number of elements of the set of firstpixels, so that the dimension-increased character vectors of thevariable number of elements of the second cluster are relatively vicinalto each other in terms of the distance in the spatiotemporal field.

Therefore, according to any genus of the invention for the third object,a flexible clustering is performed with respect to a set of characterparameters including a representative one of a relative motion estimatedbetween a sub-region of a segmented region in a current picture and acorresponding sub-region in a previous picture, with an additionallyincreased tendency to faithfully or naturally represent an originalregion connected in a picked-up image.

Accordingly, on the one hand, a number of pixels may be reclustered tobe shared among a number of resegmented regions disconnected from eachother, if their motions are significantly different from each other,even when their locations, colors and/or luminances are vicinal to eachother, thus resulting in a competent representation remarkably faithfulto an original motion image. More specifically, for example, when apicture contains a pair of neighboring objects resemblant in color butdifferent in motion, if the difference of motion is significant enoughfor an image segmentation according to the invention to separate themfrom each other, they will be distributed between a pair of disconnectedregions.

On the other hand, an image region substantially uniform in motion butuneven in color and/or luminance may be connected therein, permitting adesirable local decrease in number of regions to provide an effectiveallowance for a probable need of a minute segmentation at any otherplaces, resulting in an efficient flexible segmentation despite arestriction from a total number of regions. For example, when a picturecontains a sufficiently small set of pixels representing a single objectcolored with a pair of different colors and moving without revolution,if a resemblance between their motions is significant enough for animage segmentation according to the invention to recluster a subset ofthe pixel set with either color together with another subset with theother color, the former subset will be found in a region connected witha region including the latter subset.

It is a fourth object of the present invention to provide a method and asystem for processing a set of data on a temporal sequence of two orthree dimensional pictures with a time-dependent motion intermittentlypicked up therein, by using measures for a compensation of the motion incombination with an image segmentation of the pictures, in a feedbackingmanner so that a relative motion is estimated between a respective oneof minute-divided pieces of a current picture and a corresponding minutepiece of a previous picture to thereby determine a parameterrespresentative of the relative motion, while any piece may comprise oneor more pixels, the current picture is segmented into a number ofregions consisting of pixels clustered with respect to a set ofcharacter parameters including the motion-representative parameter, anda relative motion is reestimated between a respective one of thesegmented regions of the current picture and a correspondent region ofthe previous picture to thereby determine a motion vector.

To achieve the fourth object, a genus of the present invention providesa motion picture processing method for processing a sequence of motionpictures including a current picture and a previous picture thereto. Thecurrent picture consists of a set of current pixels. The previouspicture consists of a set of previous pixels. The current and previouspixels are each respectively defined by a combination of image dataassociated therewith and a combination of address data thereof. Themotion picture processing method comprises seven or eight steps. A firststep divides the current picture into a predetermined number of currentminute pieces each respectively consisting of one or more elements ofthe set of current pixels. A second step may divide the previous pictureinto an identical number of previous minute pieces to the predeterminednumber. The previous minute pieces may each respectively consist of oneor more elements of the set of previous pixels. A third step determinesfor a respective one of the current minute pieces a correspondingprevious minute piece. A fourth step estimates a first relative motionbetween the respective one of the current minute pieces and thecorresponding previous minute piece. A fifth step additionally defineseach element of the set of current pixels in the respective one of thecurrent minute pieces, by a combination of motion data representative ofthe first relative motion. A sixth step is for image-segmenting thecurrent picture into a number of first regions with respect to thecombination of image data, the combination of address data and thecombination of motion data, while the number of first regions is smallerthan the predetermined number. A seventh step determines for arespective one of the first regions a corresponding second region in theprevious picture. An eighth step estimates a second relative motionbetween the respective one of the first regions and the correspondingsecond region to determine a motion vector representative of the secondrelative motion.

According to a species of this genus of the invention, the motionpicture processing method further comprises the steps ofmotion-compensating a local decoded picture of the previous picture independence on the motion vector to obtain a predicted picture of thecurrent picture, subtracting the predicted picture from the currentpicture to obtain a prediction error picture, coding the predictionerror picture in a compressing manner into a set of coded data, decodingthe set of coded data in a decompressing manner into a local decodederror picture, and adding the local decoded error picture to thepredicted picture to obtain a local decoded picture of the currentpicture.

According to another species of this genus of the invention, the currentpicture comprises an arbitrary picture in the sequence of motionpictures, and the previous picture comprises a local decoded picture ofa picture previous to the arbitrary picture in the sequence of motionpictures.

To achieve the fourth object, another genus of the present inventionprovides a motion picture coding-decoding method for processing asequence of motion pictures including a current picture and a previouspicture thereto. The current picture consists of a set of currentpixels. The previous picture consists of a set of previous pixels. Thecurrent and previous pixels are each respectively defined by acombination of image data associated therewith and a combination ofaddress data thereof. The motion picture coding-decoding methodcomprises seventeen or eighteen steps. A first step divides the currentpicture into a predetermined number of current minute pieces eachrespectively consisting of one or more elements of the set of currentpixels. A second step may divide the previous picture into an identicalnumber of previous minute pieces to the predetermined number, whichprevious minute pieces may each respectively consist of one or moreelements of the set of previous pixels. A third step determines for arespective one of the current minute pieces a corresponding previousminute piece. A fourth step estimates a first relative motion betweenthe respective one of the current minute pieces and the correspondingprevious minute piece. A fifth step additionally defines each element ofthe set of current pixels in the respective one of the current minutepieces, by a combination of motion data representative of the firstrelative motion. A sixth step is for image-segmenting the currentpicture into a number of first regions with respect to the combinationof image data, the combination of address data and the combination ofmotion data, while the number of first regions is smaller than thepredetermined number. A seventh step determines for a respective one ofthe first regions a corresponding second region in the previous picture.An eighth step estimates a second relative motion between the respectiveone of the first regions and the corresponding second region todetermine a motion vector representative of the second relative motion.

A ninth step is for motion-compensating a local decoded picture of theprevious picture in dependence on the motion vector to obtain apredicted picture of the current picture. A tenth step subtracts thepredicted picture from the current picture to obtain a prediction errorpicture. An eleventh step codes the motion vector into a sequence offirst codes. A twelfth step codes the prediction error picture in acompressing manner into a sequence of second codes. A thirteenth stepmultiplexes the sequence of first codes and the sequence of second codesto obtain a multiplexed signal. A fourteenth step demultiplexes themultiplexed signal into a combination of a sequence of third codescorresponding to the sequence of first codes and a seqeunce of fourthcodes corresponding to the sequence of second codes. A fifteenth stepdecodes the sequence of third codes to obtain a decoded vectorcorresponding to the motion vector. A sixteenth step decodes thesequence of fourth codes to obtain a decoded error picture. Aseventeenth step responds to the decoded vector to motion-compensate aprevious decoded picture in a sequence of decoded pictures to obtain adecoded prediction picture. An eighteenth step adds the decoded errorpicture to the decoded prediction picture to obtain a picture subsequentto the previous decoded picture in the sequence of decoded pictures.

To achieve the fourth object, another genus of the present inventionprovides a picture data processing method for processing a set of dataon a temporal sequence of two or three dimensional pictures with atime-dependent motion intermittently picked up therein. The set of dataincludes a set of current data on a current picture in the sequence ofpictures and a set of previous data on a previous picture to the currentpicture in the sequence of pictures. The current picture consists of aset of current pixels each respectively defined by a combination ofcurrent image data associated therewith and a combination of currentaddress data representing an address thereof in the current picture. Theprevious picture consists of a set of previous pixels each respectivelydefined by a combination of previous image data associated therewith anda combination of previous address data representing an address thereofin the previous picture.

The picture data processing method comprises nine or ten steps. A firststep has the set of current data processed so that the current pictureis linearly mapped into a spatiotemporal field, as a first pictureconsisting of a set of first pixels each respectively represented by afirst character vector equivalent to a sum of a first image parametervector representative of the combination of current image data and afirst location parameter vector representative of the combination ofcurrent address data. A second step has the set of previous dataprocessed so that the previous picture is linearly mapped into thespatiotemporal field, as a second picture consisting of a set of secondpixels each respectively represented by a second character vectorequivalent to a sum of a second image parameter vector representative ofthe combination of previous image data and a second location parametervector representative of the combination of previous address data. Athird step divides the first picture into a predetermined number offirst minute pieces each respectively consisting of one or more elementsof the set of first pixels having the first location parameter vectorsthereof averaged to obtain a piece location representative vectorrepresentative of a location of a correspondent one of the first minutepieces in the spatiotemporal field. A fourth step may divide the secondpicture into an identical number of second minute pieces to thepredetermined number, which second minute pieces may each respectivelyconsist of one or more elements of the set of second pixels having thesecond location parameter vectors thereof averaged to obtain a piecelocation representative vector representative of a location of acorrespondent one of the second minute pieces in the spatiotemporalfield. A fifth step determines for a respective one of the first minutepieces a corresponding second minute piece. A sixth step estimates afirst relative motion as part of the time-dependent motion between therespective one of the first minute pieces and the corresponding secondminute piece, by subtracting a piece location representative vector ofthe corresponding second minute piece from the piece locationrepresentative vector of the respective one of the first minute piecesto determine a first motion vector representive of the first relativemotion in the spatiotemporal field. A seventh step adds the first motionvector to the first character vector of each element of the set of firstpixels in the respective one of the first minute pieces to obtain adimension-increased character vector. An eighth step segments the firstpicture into a number of first regions each respectively composed of acluster of a variable number of elements of the set of first pixels, sothat the dimension-increased character vectors of the variable number ofelements are relatively vicinal to each other in terms of a distancedefined in the spatiotemporal field and have the first locationparameter vectors thereof averaged to obtain a region locationrepresentative vector representative of a location of a correspondentone of the first regions in the spatiotemporal field. A ninth stepdetermines for a respective one of the first regions a correspondingsecond region in the second picture, which second region is composed ofan identical number of elements of the set of second pixels to thevariable number of elements of the set of first pixels, so that thesecond character vectors of the identical number of elements have thesecond image parameter vectors thereof each respectively relativelyvicinal to a representative vector of the first image parameter vectorsof the first character vectors in terms of the distance and the secondlocation parameter vectors thereof averaged to obtain a region locationrepresentative vector representative of a location of the correspondingsecond region in the spatiotemporal field. A tenth step estimates asecond relative motion as part of the time-dependent motion between therespective one of the first regions and the corresponding second region,by subtracting the region location representative vector of thecorresponding second region from the region location representativevector of the respective one of the first regions to determine a secondmotion vector representive of the second relative motion in thespatiotemporal field.

Moreover, to achieve the fourth object, another genus of the presentinvention provides a motion picture processing system for processing asequence of motion pictures including a current picture and a previouspicture thereto. The current picture consists of a set of currentpixels. The previous picture consists of a set of previous pixels. Thecurrent and previous pixels are each respectively defined by acombination of image data associated therewith and a combination ofaddress data thereof. The motion picture processing system comprises afirst motion estimator, an image segmentor and a second motionestimator. The first motion estimator divides the current picture into apredetermined number of current minute pieces each respectivelyconsisting of one or more elements of the set of current pixels, and maydivide the previous picture into an identical number of previous minutepieces to the predetermined number. The previous minute pieces may eachrespectively consist of one or more elements of the set of previouspixels. The estimator then determines for a respective one of thecurrent minute pieces a corresponding previous minute piece, estimates afirst relative motion between the respective one of the current minutepieces and the corresponding previous minute piece, and additionallydefines each element of the set of current pixels in the respective oneof the current minute pieces, by a combination of motion datarepresentative of the first relative motion. The image segmentor is forimage-segmenting the current picture into a number of first regions withrespect to the combination of image data, the combination of addressdata and the combination of motion data, while the nummber of firstregions is smaller than the predetermined number. The second motionestimator determines for a respective one of the first regions acorresponding second region in the previous picture, and estimates asecond relative motion between the respective one of the first regionsand the corresponding second region to determine a motion vectorrepresentative of the second relative motion.

According to a species of this genus of the invention, the motionpicture processing system further comprises a motion compensator formotion-compensating a local decoded picture of the previous picture independence on the motion vector to obtain a predicted picture of thecurrent picture, a subtractor for subtracting the predicted picture fromthe current picture to obtain a prediction error picture, a compressingcoder for coding the prediction error picture in a compressing mannerinto a set of coded data, a decompressing decoder for decoding the setof coded data in a decompressing manner into a local decoded errorpicture, and an adder for adding the local decoded error picture to thepredicted picture to obtain a local decoded picture of the currentpicture.

According to another species of this genus of the invention, the currentpicture comprises an arbitrary picture in the sequence of motionpictures, and the previous picture comprises a local decoded picture ofa picture previous to the arbitrary picture in the sequence of motionpictures.

To achieve the fourth object, another genus of the present inventionprovides a motion picture coding-decoding system for processing asequence of motion pictures including a current picture and a previouspicture thereto. The current picture consists of a set of currentpixels. The previous picture consists of a set of previous pixels. Thecurrent and previous pixels are each respectively defined by acombination of image data associated therewith and a combination ofaddress data thereof. The motion picture coding-decoding systemcomprises a motion estimator, an image segmentor, a motion reestimator,a first motion compensator, a subtractor, a multiplexing encoder, ademultiplexing decoder, a second motion compensator and an adder. Themotion estimator divides the current picture into a predetermined numberof current minute pieces each respectively consisting of one or moreelements of the set of current pixels, and may divide the previouspicture into an identical number of previous minute pieces to thepredetermined number, which previous minute pieces may each respectivelyconsist of one or more elements of the set of previous pixels,determines for a respective one of the current minute pieces acorresponding previous minute piece, estimates a first relative motionbetween the respective one of the current minute pieces and thecorresponding previous minute piece, and additionally defines eachelement of the set of current pixels in the respective one of thecurrent minute pieces, by a combination of motion data representative ofthe first relative motion. The image segmentor is for image-segmentingthe current picture into a number of first regions with respect to thecombination of image data, the combination of address data and thecombination of motion data, while number of first regions is smallerthan the predetermined number. The motion reestimator determines for arespective one of the first regions a corresponding second region in theprevious picture, and estimates a second relative motion between therespective one of the first regions and the corresponding second regionto determine a motion vector representative of the second relativemotion. The first motion compensator is for for motion-compensating alocal decoded picture of the previous picture in dependence on themotion vector to obtain a predicted picture of the current picture. Thesubtractor subtracts the predicted picture from the current picture toobtain a prediction error picture. The multiplexing encoder codes themotion vector into a sequence of first codes, codes the prediction errorpicture in a compressing manner into a sequence of second codes, andmultiplexes the sequence of first codes and the sequence of second codesto obtain a multiplexed signal. The demultiplexing decoder demultiplexesthe multiplexed signal into a combination of a sequence of third codescorresponding to the sequence of first codes and a seqeunce of fourthcodes corresponding to the sequence of second codes, decodes thesequence of third codes to obtain a decoded vector corresponding to themotion vector, and decodes the sequence of fourth codes to obtain adecoded error picture. The second motion compensator responds to thedecoded vector to motion-compensate a previous decoded picture in asequence of decoded pictures to obtain a decoded prediction picture. Theadder adds the decoded error picture to the decoded prediction pictureto obtain a picture subsequent to the previous decoded picture in thesequence of decoded pictures.

To achieve the fourth object, another genus of the present inventionprovides a picture data processing system for processing a set of dataon a temporal sequence of two or three dimensional pictures with atime-dependent motion intermittently picked up therein. The set of dataincludes a set of current data on a current picture in the sequence ofpictures and a set of previous data on a previous picture to the currentpicture in the sequence of pictures. The current picture consists of aset of current pixels each respectively defined by a combination ofcurrent image data associated therewith and a combination of currentaddress data representing an address thereof in the current picture. Theprevious picture consists of a set of previous pixels each respectivelydefined by a combination of previous image data associated therewith anda combination of previous address data representing an address thereofin the previous picture.

The picture data processing system comprises a set of data processingmeasures. a motion estimator and an image segmentor. The data processingmeasures serves for having the set of current data processed so that thecurrent picture is linearly mapped into a spatiotemporal field, as afirst picture consisting of a set of first pixels each respectivelyrepresented by a first character vector equivalent to a sum of a firstimage parameter vector representative of the combination of currentimage data and a first location parameter vector representative of thecombination of current address data, and for having the set of previousdata processed so that the previous picture is linearly mapped into thespatiotemporal field, as a second picture consisting of a set of secondpixels each respectively represented by a second character vectorequivalent to a sum of a second image parameter vector representative ofthe combination of previous image data and a second location parametervector representative of the combination of previous address data. Themotion estimator divides the first picture into a predetermined numberof first minute pieces each respectively consisting of one or moreelements of the set of first pixels having the first location parametervectors thereof averaged to obtain a piece location representativevector representative of a location of a correspondent one of the firstminute pieces in the spatiotemporal field, may divide the second pictureinto an identical number of second minute pieces to the predeterminednumber, which second minute pieces may each respectively consist of oneor more elements of the set of second pixels having the second locationparameter vectors thereof averaged to obtain a piece locationrepresentative vector representative of a location of a correspondentone of the second minute pieces in the spatiotemporal field, determinesfor a respective one of the first minute pieces a corresponding secondminute piece, estimates a first relative motion as part of thetime-dependent motion between the respective one of the first minutepieces and the corresponding second minute piece, by subtracting a piecelocation representative vector of the corresponding second minute piecefrom the piece location representative vector of the respective one ofthe first minute pieces to determine a first motion vector representiveof the first relative motion in the spatiotemporal field, and adds thefirst motion vector to the first character vector of each element of theset of first pixels in the respective one of the first minute pieces toobtain a dimension-increased character vector. The image segmentorsegments the first picture into a number of first regions eachrespectively composed of a cluster of a variable number of elements ofthe set of first pixels, so that the dimension-increased charactervectors of the variable number of elements are relatively vicinal toeach other in terms of a distance defined in the spatiotemporal fieldand have the first location parameter vectors thereof averaged to obtaina region location representative vector representative of a location ofa correspondent one of the first regions in the spatiotemporal field.The motion estimator is implemented for determining for a respective oneof the first regions a corresponding second region in the secondpicture, which second region is composed of an identical number ofelements of the set of second pixels to the variable number of elementsof the set of first pixels, so that the second character vectors of theidentical number of elements have the second image parameter vectorsthereof each respectively relatively vicinal to a representative vectorof the first image parameter vectors of the first character vectors interms of the distance and the second location parameter vectors thereofaveraged to obtain a region location representative vectorrepresentative of a location of the corresponding second region in thespatiotemporal field, and for estimating a second relative motion aspart of the time-dependent motion between the respective one of thefirst regions and the corresponding second region, by subtracting theregion location representative vector of the corresponding second regionfrom the region location representative vector of the respective one ofthe first regions to determine a second motion vector representive ofthe second relative motion in the spatiotemporal field.

Still more, to achieve the fourth object, another genus of the presentinvention provides a motion picture processing method for processing asequence of motion pictures including a current picture and a previouspicture thereto, by using one of the previous picture and a localdecoded picture thereof as a reference picture for an interframe motioncompensation to obtain a predicted picture of the current picture. Themotion picture processing method comprises four steps. A first stepcompares a set of current data representative of the current picturewith a set of reference data representative of the reference picture toobtain a set of differential data therebetween. A second step is forimage-segmenting the current picture with respect to a set of characterparameters thereof and the set of differential data, to obtain animage-segmented picture consisting of a number of segment regions. Athird step estimates a relative motion of a respective one of thesegment regions to a corresponding region of the reference picture. Afourth step is for motion-compensating the corresponding region of thereference picture by the relative motion to obtain a set ofmotion-compensated data representative of the predicted picture.

According to a species of this genus of the invention, the motionpicture processing method further comprises the steps of determining aset of difference data between the set of current data and the set ofmotion-compensated data, as a set of prediction error data, compressingthe set of prediction error data to obtain a set of compressed data,decompressing the set of compressed data to obtain a set of restorederror data, and adding the set of restored error data to the set ofmotion-compensated data to obtain a set of data representative of alocal decoded picture of the current picture.

According to another species of this genus of the invention, the motionpicture processing method further comprises the steps of determining aset of difference data between the set of current data and the set ofmotion-compensated data, as a set of prediction error data, compressingthe set of prediction error data to obtain a set of compressed data,coding the set of compressed data, a set of data on the imagesegmentation of the current picture and a set of data representative ofthe relative motion of the respective one of the segment regions into afirst sequence of codes, a second sequence of codes and a third sequenceof codes, respectively, decoding the first sequence of codes, the secondsequence of codes and the third sequence of codes into a first set ofdata, a second set of data and a third set of data, respectively,decompressing the first set of data to obtain a set of decompresseddata, motion-compensating a decoded picture of the previous picture byusing the second set of data and the third set of data to obtain amotion-compensated picture, and error-compensating themotion-compensated picture by said set of decompressed data to obtain adecoded picture of the current picture.

Yet more, to achieve the fourth object, another genus of the presentinvention provides a motion picture processing system for processing asequence of motion pictures including a current picture and a previouspicture thereto, by using one of the previous picture and a localdecoded picture thereof as a reference picture for an interframe motioncompensation to obtain a predicted picture of the current picture. Themotion picture processing system comprises an image segmentor, a motionestimator and a motion compensator. The image segmentor compares a setof current data representative of the current picture with a set ofreference data representative of the reference picture to obtain a setof differential data therebetween, and serves for image-segmenting thecurrent picture with respect to a set of character parameters thereofand the set of differential data, to obtain an image-segmented pictureconsisting of a number of segment regions. The motion estimatorestimates a relative motion of a respective one of the segment regionsto a corresponding region of the reference picture. The motioncompensator serves for motion-compensating the corresponding region ofthe reference picture by the relative motion to obtain a set ofmotion-compensated data representative of the predicted picture.

According to a species of this genus of the invention, the motionpicture processing system further comprises a subtractor for determininga set of difference data between the set of current data and the set ofmotion-compensated data, as a set of prediction error data, acompressing coder for compressing the set of prediction error data toobtain a set of compressed data, a decompressing decoder fordecompressing the set of compressed data to obtain a set of restorederror data, and an adder for adding the set of restored error data tothe set of motion-compensated data to obtain a set of datarepresentative of a local decoded picture of the current picture.

According to another species of this genus of the invention, the motionpicture processing method further comprises a subtractor for determininga set of difference data between the set of current data and the set ofmotion-compensated data, as a set of prediction error data, acompressing coder for compressing the set of prediction error data toobtain a set of compressed data, an encoder for coding the set ofcompressed data, a set of data on the image segmentation of the currentpicture and a set of data representative of the relative motion of therespective one of the segment regions into a first sequence of codes, asecond sequence of codes and a third sequence of codes, respectively, adecoder for decoding the first sequence of codes, the second sequence ofcodes and the third sequence of codes into a first set of data, a secondset of data and a third set of data, respectively, a decompressingdecoder for decompressing the first set of data to obtain a set ofdecompressed data, another motion compensator for motion-compensating adecoded picture of the previous picture by using the second set of dataand the third set of data to obtain a motion-compensated picture, and anerror compensator for error-compensating the motion-compensated pictureby said set of decompressed data to obtain a decoded picture of thecurrent picture.

Therefore, according to any genus of the invention for the fourthobject, a motion vector is determined by reestimating a relative motionof a cluster of pixels in a current picture, which pixels have theircharacter parameter sets each including a motion-representativeparameter, to a corresponding region of a previous picture and hence hasan additionally increased tendency to faithfully or naturally representa picked-up motion.

For example, in any segmented region, all pixels should represent asubstantially uniform motion, so that a reestimated motion of such aregion may provide a motion vector remarkably faithful to an originalimage, thus permitting an associated prediction error to be decreased.

BRIEF DESCRIPTION OF THE DRAWINGS

The objects, features and advantages of the present invention willbecome more apparent from consideration of the following detaileddescription, taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a block diagram of an encoder of a conventional motion picturecoding-decoding system using a motion compensation interframeprediction;

FIG. 2 is an illustration describing an estimation of a motion in animaginary field defined in the conventional system of FIG. 1;

FIG. 3 is a block diagram of a decoder of the conventional system ofFIG. 1;

FIG. 4 is an illustration describing a basic concept of a conventionalclustering in a dimension-reduced parameter space;

FIG. 5 is a block diagram of an exemplary motion picture processingsystem for describing a conventional image segmentation using afive-dimensional clustering;

FIG. 6 is a detailed block diagram of a clustering circuit in the blockdiagram of FIG. 5;

FIG. 7 is a block diagram of a motion picture processing system havingfour operation modes according to an embodiment of the presentinvention;

FIG. 8 is a block diagram of an encoder of the system of FIG. 7;

FIG. 9 is a block diagram of a decoder of the system of FIG. 7;

FIG. 10 is a block diagram of an image segmentation circuit of theencoder of FIG. 8;

FIG. 11 is a block diagram of an image segmentation circuit according toanother embodiment of the present invention;

FIG. 12 is a block diagram of an image segmentation circuit according toanother embodiment of the present invention;

FIG. 13 is a block diagram of an image segmentation circuit according toanother embodiment of the present invention;

FIG. 14 is a block diagram of an image segmentation circuit according toanother embodiment of the present invention;

FIG. 15 is a block diagram of an image segmentation circuit according toanother embodiment of the present invention;

FIG. 16 is a block diagram of an image segmentation circuit according toanother embodiment of the present invention; and

FIG. 17 is an illustration describing a basic concept of a mappingaccording to the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

There will be detailed below preferred embodiments of the presentinvention, with reference to the accompanying drawings. Like orcorresponding items are designated by like or corresponding charactersfor brevity.

FIG. 7 shows an entirety of a motion picture processing system accordingto an embodiment of the invention.

In FIG. 7, designated at character 1 is the motion picture processingsystem. The system 1 serves for processing a temporal sequence of two orthree dimensional motion pictures with a time-dependent motion of anobject 2 intermittently picked up therein. Any picture in the picturesequence appears in a variety of forms as it is processed, for example,as a set of optical signals A2, a sequence of analog signals A3 or A6, asequence or set of digital data D4 or D20, or a sequence or set of codesC10 or C11.

The system 1 includes a camera 3 for converting the optical signals A2to the digital signals A3, an analog to digital converter 4 forconverting the analog signals A3 to the digital data D4, and an encoder10 for converting the digital data D4 to the codes C10 or C11. Somecodes C10 are storaged in a storage medium 5, to be read therefromlater. Some codes C11 are transmitted via a transmission line 22. Thesystem 1 further includes a decoder 30 for converting the codes C10 orC11 to the digital data D20, a digital to analog converter 6 forconverting the digital data D20 to the analog signals A6, and a monitor7 for displaying the sequence of pictures.

FIG. 8 is a block diagram of the encoder 10, and FIG. 9 is a blockdiagram of the decoder 30.

As shown in FIG. 8, the encoder 10 comprises an input terminal 11, aframe memory 12, a motion estimator 13, a motion compensator 14, asubtractor 15, a compression circuit 16, a decompression circuit 17, anadder 18, an encoding multiplexer 20, an output terminal 21, and animage segmentation circuit 23 which will be detailed later.

As shown in FIG. 9, the decoder 30 comprises an input terminal 36, adecoding demultiplexer 31, a decompression circuit 32, a motioncompensator 33, an adder 34, a frame memory 35 and an output terminal37.

The encoder 10 as well as the decoder 30 has various unshown dataprocessing programs thereof stored therein or in an unshown controllerof the system 1, so that any picture in the picture seqeunce is mappedin an imaginary spatial field and/or an imaginary spatiotemporal field,when a set of associated data is processed.

Incidentally, FIG. 17 illustrates a basic concept of the mapping.

As shown in FIG. 17, any and all picture Pc at a time t is congruentlymapped in a two (or three) dimensional spatial field defined by an X-Y(or X-Y-Z) coordinate system, when a set of associated data is processedby the programs, and any and all picture Pc as a set of pixel characterdata Px is linearly mapped therefrom to a multi-dimensionalspatiotemporal field defined by a φ-T coordinate system, i.e. a φ_(x)-φ_(y) -φ_(a) -T system (or a φ_(x) -φ_(y) -φ_(z) -φ_(a) -T system),when any and all associated character data is processed by the programs.The spatiotemporal field and its relationship with pixel characterparameters will be detailed later. A set of processed results in thespatiotemporal field is inverse linear mapped into the spatial field,wherefrom a set of associated data is inverse mapped or output to a realspace in concern.

As shown in FIG. 8, the input terminal 11 of the encoder 10 inputs asequence of pixel data D11 of a current picture P11. The frame memory 12stores therein the data D11 of the current input picture P11, besides aset of data D10 of a past input picture P10 it received from the inputterminal 11 and stored therein in a last frame. The frame memory 19 hasstored therein a set of data D12 of a local decoded picture P12 that isa matrix of restored pixel data of the past input picture P10.

The data D10 of the past input picture P10 are sequentially read fromthe frame memory 12, and the data D12 of the local decoded picture P12from the frame memory 19. They are either selected by a switch 13a aswell as by a switch 23a, as a sequence of data D13 representing areference picture P13 (to be P10 or P12), and input to the motionestimator 13 which concurrently receives the data D11 of the currentinput picture P11, as well as to the image segmentation circuit 23 whichalso concurrently receives the data D11 of the current input pictureP11.

The image segmentation circuit 23 executes an image segmentation of thecurrent picture P11, and outputs a set of data D23 on a result RSthereof.

The motion estimator 13 estimates, componentwise of character parameter,a relative motion between the current picture P11 and the referencepicture P13 to determine a motion vector VM representative of therelative motoin, and outputs a set of data D14 thereon.

For a feedback connection from the motion estimator 13 to the imagesegmentation circuit 23, a feedback control switch 23b is installedtherebetween.

Incidentally, the system 1 has four different modes of operation thatare selective thereamong in accordance with a requisite display quality.

In a first mode for an improved intermediate quality, the currentpicture P11 is segmented into a number of regions as clusters of pixels,and a relative motion is estimated between a respective one of thesegmented regions in the current picture P11 and a corresponding regionof the reference picture P13 to thereby determine a motion vector.

In a second mode for an increased intermediate quality, a relativemotion is estimated between a respective one of minute-divided pieces ofthe current picture P11 and a corresponding minute piece in thereference picture P13 to thereby determine a parameter respresentativeof the relative motion, while any piece may comprise one or more pixels,and the current picture P11 is segmented into a number of regionsconsisting of pixels clustered with respect to a set of characterparameters including the motion-representative parameter.

In a third mode for an improved high quality, the current picture P11 issegmented into a number of regions as clusters of pixels, a relativemotion is estimated between a respective one of sub-regions of arespective one of the segmented regions in the current picture P11 and acorresponding small region in the reference picture P13 to therebydetermine a representative parameter of the relative motion, while anysub-region may comprise one or more pixels, and the current picture P11is resegmented into a number of regions consisting of pixels reclusteredwith respect to a set of character parameters including themotion-representative parameter.

In a most complicated fourth mode for an increased high quality, arelative motion is estimated between a respective one of minute-dividedpieces of the current picture P11 and a corresponding minute piece ofthe reference picture P13 to thereby determine a parameterrepresentative of the relative motion, while any piece may comprise oneor more pixels, the current picture P11 is segmented into a number ofregions consisting of pixels clustered with respect to a set ofcharacter parameters including the motion-representative parameter, anda relative motion is reestimated between a respective one of thesegmented regions of the current picture P11 and a correspondent regionof the reference picture P13 to thereby determine a motion vector.

The image segmentation circuit 23, the motion estimator 13 and thefeedback control switch 23b as well as other associated circuits in theencoder 10 and the decoder 30 are controlled in accordance with aselected one of the four modes.

The fourth mode will be described. For an intuitive comprehension, thereference picture P13 will be substituted by a previous picture to thecurrent picture P11 in the motion picture sequence, as it is asingle-frame delayed picture to the current picture P11 and as a matterof course one of past input pictures, and each minute piece will beconstituted by a single pixel.

In the fourth mode, the motion estimator 13 first serves to determine amotion vector representative of a relative motion of a respective one ofpixels Px in the current picture P11 to a corresponding pixel in theprevious picture P13. The correspondence is determined by comparing anexpanded small region of each pixel Px with a respective one ofminute-divided regions of the same figure in the previous picture tominimize a relative distance between representative vectors thereof inthe spatiotemporal field. Each representative vector may represent acombination of averaged parameter values of colors and/or locations.

Then, the image segmentation circuit 23 clusters the pixels Px into apredetermined number of clusters, by using a combination of image data,a location in the picture P11 and the relative motion of the respectivepixel Px as a set of character parameters of that pixel Px. The imagesegmentation circuit 23 may have an exclusive motion estimator therefor.Respective pixels Px clustred in a cluster have their sets φ' ofcharacter parameters φ_(b) elementwise relatively vicinal to each other,including location parameters φ_(x), φ_(y) (and φ_(z)), and areneighboring each other, thus constituting a connected region. Each pixelPx is labelled with an identification number of the connected region inthe current picture P11. The labelled number is information associatedwith the pixel Px, as a result RS of image segmentation, which is outputto to the motion estimator 13, the motion compensator 14 and theencoding multiplexer 20. The multiplexer 20 encodes the data D14 ofmotion vectors VM into a sequence of corresponding codes.

The motion estimator 13 receives the data D11 of the current pictureP11, the data D13 of the previous picture P13 and the data D23 on theresult RS of the image segmentation based thereon, i.e. on the data D11and D13, and estimates a relative motion of a respective one ofimage-segmented regions of the current picture P11 to a correspondingregion in the previous picture P13 to determine a motion vector VMrepresentative of the relative motion. Any motion is defined in thespatiotemporal field, as a distance between vector-represented pointsand may be increased with time if either point be fixed, while a signalsynchronization or a frame selection permits a normal use of a fixedtemporal component.

To determine the motion vector VM as a displacement vector of a region,there may be employed a similar concept to the conventional blockmatching. The result RS of image segmentation may thus be employed tohave an image-segmented region as a criterion in place of anequi-divided block for a matching detection. A matching may be detectedby determining a target region at a corresponding location in theprevious picture P13, before shifting the target region within adetection area of the previous picture P13 to detect a location at whicha character value differernce of image data to the current picture P11is minimized. In that case, a set of X-Y (or X-Y-z) componentwiseexpressed motion vectors VM be output from the estimator 13.

In another method of motion estimation, the relative motion may beexpressed by an affinity, which may permit a combination of affinetransform parameters. For example, a certain pixel Px(T-1) may have alocation (x,y) in the previous picture P13, and another pixel Px(T) mayhave a different location (x',y') in the current picture P11. Thelocations (x,y) and (x',y') may be each expressed by a 2-line, 1-columnvector, respectively, to provide an equation therebetween in which thevector of location (x,y) is affine transformed by a combination of afactor consisting of a 2-line, 2-column vector of four transformparameters and a term consisting of a 2-line, 1-column vector of twotransform parameters. As the location (x,y) is shifted within adetection area, the six parameters may be varied so that a predictionerror is minimized or decreased from a preset value to a requisitevalue, with a combination of parameter values to determine a location inconcern. In such a case, a combination of affine transform parametersmay be output from the estimator 13.

The data D14 of motion vectors VM are sequentially output from themotion estimator 13 to the motion compensator 14 and the encodingmultiplexer 20.

The multiplexer 20 encodes the data D14 of motion vectors VM into asequence of corresponding codes.

At the motion compensator 14, the data D14 of motion vectors VM areprocessed together with the result RS of image segmentation and the dataD12 of the local decoded picture P12 input from the frame memory 19, tomake a motion compensation of each segmented region.

In the case of a componentwise represented motion vector of a segmentedregion, a combination of data of a corresponding pixel in position inthe local decoded picture P12 is taken as data of a pixel at a motioncompensated location.

In the case of a combination of affine transform parameters, acombination of pixel data at an affine-expressed location in the localdecoded picture P12 is taken as data of a pixel at a correspondentlocation in the current picture P11.

As a result, each pixel data in each region is updated by a data that acorresponding pixel in a corresponding region in the local decodedpicture P12 had been carrying.

The motion compensator 14 sequentially outputs a set of data D15 of themotion-compensated picture P14 to a subtractor 15 and an adder 108.

The subtractor 15 performs a pixel-mode subtraction of themotion-compensated picture P14 from the current input picture P11,obtaining a set of data D16 respresentative of a differential pictureP15 therebetween. Accordingly, the data D11 of the current input pictureP11 are converted into a compressed set of data as the data D16respresenting prediction errors.

The prediction error data D17 of the differential picture P15 aresequentially input from the subtractor 15 to the compression circuit 16as a unit 16 adapted for a discrete cosine transformation andquantization process for a data compression. As a result, the error dataD16 are further compressed into a set of combinations of data D17 eachrepresentative of a quantized coefficient, so that the set of data D17represents the differential picture P15. In place of DCT-Q, there may beemployed a Wavelet transform or an Hadamard transform and a vectorquantization, fractal coding or intra-frame prediction coding, alone orin combination.

The compressed data D17 of the differential picture P15 are sequentiallyoutput from the compression circuit 16 to the encoding multiplexer 20and the decompression circuit 17 as a unit adapted for an inversequantization and inverse discrete cosine transformation process, for adata decompression.

The multiplexer 20 encodes the compressed data D17 of the differentialpicture P15 into a sequence of corresponding codes, and multiplexes themtogether with the codes of the result RS of image segmentation and thecodes of the the motion vectors VM, into a sequence of multiplexed codesC11 to be output via the output terminal 21 of the encoder 10.

At the IQ-IDCT unit 17, each input combination of data D17 is inversequantized into a combination of corresponding cosine coefficients of acosine series in a related frequency field, which coefficients are theninverse mapped from the frequency field through an inverse discretecosine transform function into the real measure space, as a set of dataD19.

The data D19 of a local decoded or restored differential picture P16 aresequentially output to the adder 18, where they are subjected to apixel-mode addition with the data D15 of the motion-compensated pictureP14 input from the motion compensator 14, to thereby obtain a set ofdata D21 representative of a local decoded or restored picture P17equivalent to the current input picture P11.

The data D21 of the restored current picture P17 is sequentially inputfrom the adder 18 to the frame memory 19, where they are stored atcorresponding addressed locations, as a set of data representing a localdecoded current picture to be employed, in a subsequent frame, as asubsequent local decoded picture of a subsequent previous picture.

The sequence of output codes C11 is transmitted through the transmissionline 22 to the decoder 30.

The demultiplexer 31 demultiplexes the codes C11 into a first codesequence representative of the prediction error picture P15, a secondcode sequence representative of the result RS of image segmentation anda third code sequence representative of motion vectors VM, and decodesthe first code sequence into a sequence of data D31 equivalent to thecompressed data D17, the second code sequence into a sequence of dataD36 equivalent to the data D23, and the third code sequence into asequence of data D32 equivalent to the data D14.

The data D31 are input to the decompression circuit 32 as an IQ-IDCTunit, which functions in a similar manner to the IQ-IDCT unit 17, thussequentially outputting a set of data D33 representative of adifferential picture P31 equivalent to the restored differential pictureP16.

The data D36 and D32 are input to the motion compensator 33, whichconcurrently receives a set of data D30 of a decoded picture P30 of theprevious picture P13 from the frame memory 35 and compensates these dataD30 by those data D36 and D32 in a similar manner to the motioncompensator 14, thus sequentially outputting a set of data D34representative of a motion-compensated picture P32 as a predictedcurrent picture equivalent to the motion-compensated picture P14.

The data D32 of the decoded differential picture P31 and the data D34 ofthe motion-compensated picture P32 are input to an adder 34, where theyare added to each other in a similar manner to the adder 18, to therebyobtain a set of data D35 representative of a current decoded picture P33equivalent to the local decoded picture P17, which data are output as adatastream via the output terminal 37 of the decoder 100b. Thisdatastream is branched to be input to the frame memory 35, where it isstored as a set of data reprentative of the current decoded picture P33to be employed as a subsequent previous decoded picture in thesubsequent frame.

FIG. 10 shows a block diagram of the image segmentation circuit 23.

The image segmentation circuit 23 comprises an address generator 40, aframe memory 41, another frame memory 42, a motion vector memory 43 anda clustering circuit 44.

The data D11 of the current picture P11 are stored in the frame memory41, and the data D13 of the previous picture P13 are stored in the framememory 42.

The data D11 and D13 are sequentially read to be input to the motionestmator 13, where they are employed to determine a motion vector(Vx,Vy) of each pixel Px between frames, i.e. between the currentpicture P11 and the previous picture P13.

The motion estimation may be by a block matching method. Letting (x,y)be a location of pixel in the curernt frame picture P11 and ±dx and ±dybe horizontal and vertical extensions therefrom to have an expandedsquare region thereabout, respectively, a difference diff(vx,vy) betweenthe current picture and the previous picture is calculated for apredetermined detection range on the previous picture to determine amotion vector that minimizes the differential, as a square region ismoved about a corresponding location within the predetermined range,such that: ##EQU6## where, S_(ref) (x,y) is a pixel image data of thecurrent picture and S_(cur) (x,y) is a pixel image data of the previouspicture.

The difference diff(vx,vy) may be such that: ##EQU7##

Such the difference is calculated over the predetermined range todetermine a location where it is minimized, which location has adirectional relative distance to the pixel Px in concern in the currentpicture P11, which distance is componentwise output to be detected asthe motion vector (Vx,Vy).

The motion vector is stored in the motion vector memory 43, or morespecifically, componentwise in a memory 43a for a horizontal componentVx and in a memory 43b for a horizontal component Vy.

The address generator 40 generates a set of address data D40a and D40bfor reading corresponding pixel data D41 from the memory 101 andcomponent data D43a and D43b of a corresponding motion vector VM. Theaddress generator 40 further generates a combination of data D40c on alocation (x,y) of a corresponding pixel in the current picture P11.

The clustering circuit 44 serves for a clustering to execute the imagesegmentation by using the pixel data D41 of the current picture P11 fromthe memory 41, the component data D43a and D43b from the memory 43 andthe location data D40c from the address generator 40.

A number of initial clusters are set by equi-dividing the picture P11into a number of square blocks, which are represented by a set {C_(i)(i=1, 2, . . . , n)} of cluster vectors each respectively representing acombination of image data, a combination of location data and the motionvector of a pixel at a geometrically central position of the cluster,C_(i) is a five-dimensional vector such that C_(i) =(Sc_(i), xc_(i),yc_(i), Vxc_(i), Vyc_(i)).

Then, a respective one P of the pixels Px in the current picture P11 hasa relative distance thereof calculated to a respective one of theclusters, and is clustered to a most vicinal cluster. Letting (x_(p),y_(p)), S_(p) and (Vx_(p), Vy_(p)) be a location, image data and motionvector of the pixel P, the respective pixel P has a parameter vectorX_(p) representative of a set of character parameters such that X_(p)=(S_(p), x_(p), y_(p), Vx_(p), Vy_(p)).

The relative distance of the pixel P to a cluster represented by avector C_(i) is defined such that: ##EQU8## where c₁ to c₆ are eachrespectively a weighting factor for a corresponding one of the characterparameters.

Then, respective pixels clustered to a respective one of the clustershave image data and location data thereof componentwise averaged toprovide the cluster with an updated representative vector C_(i)=(Sc_(i), xc_(i), yc_(i), Vxc_(i), Vyc_(i)), such that: ##EQU9## whereN_(i) is a number of the respective pixels, and S_(m), x_(m), y_(m),Vx_(m) and Vy_(m) are data values of a respective one of the pixels.

Like clustering is repeated until such the representative vector is keptunchanged by the repeating operation. Finally, all the pixels grouped ina cluster are connected thereamong, thus constituting a connected regionin which pixels are relatively vicinal to each other in location and invalue of image data.

Such the clustering may be achieved in a similar manner to the circuitof FIG. 6, subject to the use of a motion vector as a characterparameter of pixel. In application to FIG. 6, the distance calculator231 may employ the expression (24) or (25) , and average calculator 237may employ the expressions (26) to (30).

The weighting factors c_(i) in the expressions (5) and (6) have theirvalues controlled in dependence on which character parameter is moreessential to the clustering. For example, c₁ is increased if the imagedata are relatively imporant, C₄ and c₅ if the location data are moreimportant, and c₆ and c₇ if the motion data are. A dynamic range ofsignal value may be regulated by a setting. Such values may be allpreset. In the mapping to a spatiotemporal field, each weighting factorc_(i) appears as a square root of a scale factor of coordinate axis.

The relative motion of pixel may be estimated by a gradient method or anoptical flow method.

FIG. 11 shows an image segmentation circuit according to anotherembodiment of the invention, as a modification of the circuit of FIG.10.

In FIG. 11, each set of image data comprises a combination of colordata, i.e. a luminance data Y and a pair of color difference data Cb andCr.

Data D11 of a current picture P11 are sequentially stored in a framememory 50, or more specifically, a Y (luminance) data D51, a Cb (colordifference) data D52 and a Cr (color difference) data D53 of the picturedata D11 are stored in blocks 51, 52 and 53 of the memory 50,respectively. Data D13 of a previous picture P13 are stored in a framememory 42. The Y data D51 of the current picture P11 and a luminancedata D42 of the previous picture P13 are read to be input to a motionestimator 13, where they are employed to determine a horizontalcomponent Vx and a vertical component Vy of a motion vector VM of eachpixel Px between frames. A set of data D13 on the motion vector VM isstored in a motion vector memory 43 in a pixel mode and componentwise,so that a data D43a on the horizontal component Vx and a data D43b onthe vertical component Vy of each pixel are stored at addresses of thepixel in blocks 43a and 43b of the memory 43, respectively.

An address generator 54 generates a combination of address data D54a forreading image data D50 from the memory 50 and a combination ofcorresponding address data D54b for reading a set of data D43 on themotion vector VM from the memory 43. The address generator 54 furthergenerates a combination of data D54c on a location (x,y) of acorresponding pixel in the current picture P11.

A clustering circuit 44 receives the Y data D51 and Cb and Cr data D52and D53 from the memory 50, the Vx and Vy component data D43a and D43bfrom the memory 43, and the location data D54c from the addressgenerator 54, and serves for a clustering in a seven-dimensionalcharacter parameter field.

In the clustering, a relative distance d(X_(i), C_(p) is calculatedbetween a character parameter vector Xp=(Y_(p), Cb_(p), Cr_(p), x_(p),y_(p), Vx_(p), Vy_(p)) of a respective one P of pixels Px in the currentpicture P11 and a representative vector C_(i) =(Yc_(i), Cbc_(i),Crc_(i), xc_(i), yc_(i), Vxc_(i), Vyc_(i)) of respective one C_(i) ofclusters, such that: ##EQU10## where c₁ to C₇ are each respectively aweighting factor for a corresponding one of the character parameters.

Then, respective pixels clustered to a respective one of the clustershave image data and location data thereof componentwise averaged toprovide the cluster with an updated representative vector C_(i)=(Yc_(i), Cbc_(i), Crc_(i), xc_(i), yc_(i), Vxc_(i), Vyc_(i)), suchthat: ##EQU11## where N_(i) is a number of the respective pixels, andY_(m), Cb_(m), Cr_(m), x_(m), y_(m), Vx_(m) and Vy_(m) are data valuesof a respective one of the pixels.

Like clustering may be achieved in a similar manner to the circuit ofFIG. 6, subject to the use of a seven-dimensional character parametervector. In application to FIG. 6, the distance calculator 231 may employthe expression (31) or (32) , and average calculator 237 may employ theexpressions (33) to (39).

In the above embodiment, the combination of Y, Cb and Cr data may bereplaced by a combination R (red), G (green) and B (blue) color data.

In that case, one of the three color data stands for the Y data. Forexample, Y data may be substituted by the G data which is mostcorrelevant thereto among the three color data.

In a clustering, a relative distance d(X_(i), C_(p)) may thus becalculated between a character parameter vector X_(p) =(R_(p), G_(p),B_(p), x_(p), y_(p), Vx_(p), Vy_(p)) of a respective one P of pixels Pxin the current picture P11 and a representative vector C_(i) =(Rc_(i),Gc_(i), Bc_(i), xc_(i), yc_(i), Vxc_(i), Vyc_(i)) of respective oneC_(i) of clusters, such that: ##EQU12## where c₁ to C₇ are eachrespectively a weighting factor for a corresponding one of the characterparameters.

Then, respective pixels clustered to a respective one of the clustershave image data and location data thereof componentwise averaged toprovide the cluster with an updated representative vector C_(i)=(Rc_(i), Gc_(i), Bc_(i), xc_(i), yc_(i), Vxc_(i), Vyc_(i)), such that:##EQU13## where N_(i) is a number of the respective pixels, and R_(m),G_(m), B_(m), X_(m), Y_(m), Vx_(m) and Vy_(m) are data values of arespective one of the pixels.

FIG. 12 shows an image segmentation circuit according to anotherembodiment of the invention, as a modification of the circuit of FIG.11.

In FIG. 12, each set of image data comprises a combination of colordata, i.e. a luminance data Y and a pair of color difference data Cb andCr.

Data D11 of a current picture P11 are sequentially stored in a framememory 50, or more specifically, a Y (luminance) data D51, a Cb (colordifference) data D52 and a Cr (color difference) data D53 of the picturedata D11 are stored in blocks 51, 52 and 53 of the memory 50,respectively. Data D13 of a previous picture P13 are stored in a framememory 60. The Y, Cr and Cb data D51, D52 and D53 of the current pictureP11 and a Y data D61, Cr data D62 and Cb data D63 of the previouspicture P13 are read to be input to a motion estmator 64, where they areemployed to determine a motion vector VM of each pixel Px betweenframes.

In this motion estimation, a block matching is detected between arespective one of current character parameters Y, Cr and Cb and acorresponding one of previous character parameters Y, Cr and Cb, todetermine a difference diff_(y), diff_(cb) or diff_(cr) therebetween.Then, respective differences are added for estimation of a relativemotion to determine a combination of horizontal and vertical componentsof a motion vector VM in accordance with the expression (22) or (23),such that:

    diff=diff.sub.y +diff.sub.cb +diff.sub.cr                  (49).

A precise motion detection is permitted with an increased number ofcharacter parameters.

A set of data D64 on the motion vector VM is stored in a motion vectormemory 43 in a pixel mode and componentwise. Thereafter, a clustering isperformed in a seven-dimensional character parameter field.

In the above embodiment, the combination of Y, Cb and Cr data may bereplaced by a combination R (red), G (green) and B (blue) color data.

In that case, three differences diff_(R), diff_(G) and diff_(B) of R, Gand B data may be calculated and added for a motion estimation, suchthat:

    diff=diff.sub.R +diff.sub.G +diff.sub.B                    (50).

FIG. 13 shows an image segmentation circuit according to anotherembodiment of the invention, as a modification of the circuit of FIG.10.

In the arrangement of FIG. 13, a weighting controller 70 is added. Thecontroller 70 controls a weighting factor for use in a distancecalculation in clustering, in dependence on a motion in picture. Forexample, if a set of measured motions in a picture is relativelysignificant, the weighting factor is increased for a better structuralanalysis of the picture. The degree of significance may be determined interms of an averaged magnitude of motion vector per pixel, such that:##EQU14## where, Vav is an average vector, N is a total number of pixelsin a picture, and Vx and Vy are components of a motion vector of a pixelP.

Then, a weighting control is performed by using a preset constantC_(color) as a luminance-color weighting constant, such that:

    C.sub.1 =C.sub.2 =C.sub.3 =C.sub.color                     (52)

and

    C.sub.6 =C.sub.7 =K·Vav·C.sub.color      (53),

where K is a proportion constant responsive for a control strength.

FIG. 14 shows an image segmentation circuit according to anotherembodiment of the invention, as a modification of the circuit of FIG.10.

The circuit of FIG. 14 additionally includes a minute region eliminator80 for eliminating various minute regions due to a clustering.

Each region segmented by the clustering is checked for the number ofpixels to be equal to or larger than a threshold value and for arelation to each neighboring region. If the number of pixels in a regionis smaller than the threshold value, the region is judged to be minuteand integrated with a resemblant neighboring region. For the judgment ofresemblance, a distance is calculated between character parametervectors representative of regions in concern, by using expressions (24),(31) and (40) or (25), (32) and (41). Any minute region is thusintegrated with a region of which the representative vector is mostvicinal to the representative vector the minute region.

FIG. 15 shows an image segmentation circuit according to anotherembodiment of the invention, as a modification of the circuit of FIG.10.

The circuit of FIG. 15 additionally includes a noise filter 90 forfiltering noises from image data D90 of a current picture P11 and a pairof noise filters 91a and 91b for filtering noises from component dataD43a and D43b of a motion vector VM. Otherwise, such noises may cause ameaningless segmentation, resulting in unexpected minute regions.

The filters should be effective to pass significant changes along aboundary of each region, and may comprise a median filter or modefilter.

In use of a median filter, a pixel in concern in the picture is coveredwith a square area extended therearound in a horizontal direction bywidths of ±dx and in a vertical direction by widths of ±dy, to determinea central one of data values of pixels in the square area, as they arearranged in an order of magnitude.

In the case of a mode filter, a pixel in concern in the picture iscovered with a similar square area, to determine a highest one ofoccurrence frequencies of data values of pixels in the square area.

FIG. 16 shows an image segmentation circuit according to anotherembodiment of the invention, as a modification of the circuits of FIGS.10 to 15.

The circuit of FIG. 16 includes respective components employed in thecircuits of FIGS. 10 to 15.

In any embodiment described, an image segmentation may be performed byclustering pixels with respect to a combination of image data and motiondata, without the use of location data. In such a segmentation,weighting factors C₄ and c₅ may be set to a null.

As detailed above for the embodiment of FIGS. 8 to 10 and variousmodifications thereof, the fourth mode includes seven or eightfundamental steps.

A first step divides the current picture P11 into a predetermined numberof current minute pieces each respectively consisting of one or morepixels Px. A second step may divide the previous picture P13 into thesame number of previous minute pieces each respectively consisting ofone or more pixels Px. A third step determines for each current minutepiece a corresponding previous minute piece. A fourth step estimates foreach current minute piece a relative motion rm thereof to thecorresponding previous minute piece.

The second to fourth steps may be concurrently performed in a combinedmanner. A fifth step provides each pixel in every current minute piecewith a combination of motion data d14 representative of the relativemotion rm, as additional character data φ_(i).

A sixth step image-segments the current picture P11 into a number ofregions of pixels Px clustered with respect to the image data, addressdata and motion data. The number of the regions is smaller than that ofthe minute pieces.

A seventh step determines for each region of the current picture P11 acorresponding region in the previous picture P13. An eighth stepreestimates for each region of the current picture P11 a relative motionRM thereof to the corresponding region in the previous picture P13 todetermine a motion vector VM representative of this relative motion RM.

In other words, in the fourth mode, the motion estimator 13 divides thecurrent picture P11 into a predetermined number of current minute pieceseach respectively consisting of one or more pixels Px, and in some casesmay divide the previous picture P13 into an identical number of previousminute pieces each respectively consisting of one or more pixels Px.

The estimator 13 then determines for each current minute piece acorresponding previous minute piece, estimates for each current minutepiece a relative motion rm thereof to the corresponding previous minutepieces, and provides each pixel Px in the current minute piece with acombination of motion data d14 representative of the relative motion rm,which combination of motion data d14 is output via the mode controlswitch 23b to the image segmentation circuit 23 together with associatedaddress data (i.e. in synchronizm with an address supplied from anunshown address generator).

In a modified case, the estimator 13 may provide each pixel Px in theprevious minute piece with a combination of motion data d14representative of the relative motion rm, which combination of motiondata d14 may be output via the mode control switch 23b to the imagesegmentation circuit 23 together with associated address data.

The image segmentation circuit 23 serves for image-segmenting thecurrent picture P11 into a number of regions of pixels Px clustered withrespect to the image data, address data and motion data, so that eachpixel Px in the current picture P11 is labelled with an identificationnumber (hereafter "ID") of an associated region, while the region ID isoutput as the result RS of estimation in a form of data D23 combinedwith an associated pixel address to the motion estimator 13, motioncompensator 14 and encoding multiplexer 20.

In the modified case, the image segmentation circuit 23 may serve forimage-segmenting the previous picture P13 into a number of regions ofpixels Px clustered with respect to the image data, address data andmotion data, so that each pixel Px in the previous picture P13 may belabelled with an ID of an associated region. The region ID may be outputas the result RS of estimation in a form of data D23 combined with anassociated pixel address.

Then, the motion estimator 13 determines for each region of the currentpicture P11 a corresponding region in the previous picture P13, andreestimates for that region a relative motion RM thereof to thecorresponding region to determine a motion vector VM representative ofthe reestimated relative motion RM.

In the modified case, the motion estimator 13 may determine for eachregion of the previous picture P13 a corresponding region in the currentpicture P11, and reestimates for that region a relative motion RMthereof to the corresponding region to determine a motion vector VMrepresentative of the reestimated relative motion RM.

The motion compensator 14 motion-compensates the local decoded pictureP12 of the previous picture P13 by using the result RS of imagesegmentation and the motion vector VM, to obtain a predicted picture P14of the current picture P11, so that pixels in the picture P12 areclustered in accordance with the result RS of image segmentation anddisplaced in accordance with the motion vector VM together with theirimage data. The local decoded picture P12 may be image-mosaicked so thatall pixels in each cluster has a combination of averaged image data, orin some case may be simply equi-divided into blocks to bemotion-compensated by using the motion vector VM only.

The subtractor 15 subtracts the predicted picture P14 from the currentpicture P11 to obtain a prediction error picture P15. The compressioncircuit 16 codes the prediction error picture P15 in a compressingmanner into a set of coded data D17. The decompression circuit 17decodes the set of coded data D17 in a decompressing manner into a localdecoded error picture P16. The adder 18 adds the local decoded errorpicture P16 to the predicted picture P14 to obtain a local decodedpicture P17 of the current picture P11.

The encoding multiplexer 20 codes the data D14 of the motion vector VMand the data D23 of the result RS of estimation into sequences ofcompressed codes and the coded data D17 of the prediction error pictureP15 into a sequence of yet compressed codes, and multiplexes thosecompressed codes to obtain a multiplexed signal C11. The decodingdemultipler 31 demultiplexes the multiplexed signal C11 into sequencesof compressed codes and a sequence of double compressed codes, anddecodes the former into a set of data D36 representative of a decodedsegmentation corresponding to the result RS of segmentation and a set ofdata D32 representative of a decoded vector corresponding to the motionvector VM, and the latter into a set of data D31 representative of adecoded error picture corresponding to the prediction error picture P15.The motion compensator 33 employs the data D36 and D32 tomotion-compensate a previous decoded picture P30 in a similar manner tothe motion compensation at the motion compensator 14. The decompressioncircuit 32 decodes the data D31 in a decompressing manner to obtain aset of data D33 representative of a decoded error picture P31corresponding to the local decoded error picture P16. The adder 34 addsthe decoded error picture P31 to the decoded prediction picture P32 toobtain a decoded current picture P33.

In any mode, a set of current data is processed by programs in thesystem so that the current picture P11 is linearly mapped into thespatiotemporal field φ-T. as a first picture Pc(T) consisting of a setof first pixels Px(T) each respectively represented by a first charactervector V1 equivalent to a vector sum of a first image parameter vectorV11 representative of the combination of current image data φ_(a) and afirst location parameter vector V12 representative of the combination ofcurrent address data φ_(x), φ_(y) (and _(z)). Likewise, a set ofprevious data is processed so that the previous picture P13 is linearlymapped into the spatiotemporal field φ-T. as a second picture Pc(T-1)consisting of a set of second pixels Px(T-1) each respectivelyrepresented by a second character vector V2 equivalent to a vector sumof a second image parameter vector V21 representative of the combinationof previous image data φ_(a) and a second location parameter vector V22representative of the combination of previous address data φ_(x), φ_(y)(and φ_(z)). Moreover, another set of previous data is processed so thatthe local decoded previous picture P12 is linearly mapped into thespatiotemporal field φ-T, as a third picture Pc'(T-1) consisting of aset of third pixels Px'(T-1) each respectively represented by a thirdcharacter vector V3 equivalent to a vector sum of a third imageparameter vector V31 representative of a combination of local decodedprevious image data φ_(a) and a third location parameter vector V32representative of a combination of local decoded previous address dataφ_(x), φ_(y) (and φ_(z)).

In the fourth mode, the motion estimator 13 divides the first picturePc(T) into Mp×Np (×Rp) (Mp, Np, Rp=predetermined integers) first minutepieces Pp(T) each respectively consisting of one or more first pixelsPx(T) having the first location parameter vectors V12 thereof averagedto obtain a piece location representative vector V1p representative of alocation (φ_(x), φ_(y) (and φ_(z))) of the first minute piece Pp(T) inthe spatiotemporal field, and in some case may divide the second picturePc(T-1) into Mp×Np (×Rp) second minute pieces Pp(T-1) each respectivelyconsisting of one or more second pixels Px(T-1) having the secondlocation parameter vectors V22 thereof averaged to obtain a piecelocation representative vector V2p representative of a location (φ_(x),φ_(y) (and φ_(z))) of the second minute piece Pp(T-1) in thespatiotemporal field. The estimator 13 determines for each first minutepiece Pp(T) a corresponding second minute piece Pp(T-1), and estimates adifference vector V1p-V2p therebetween, as a first relative motion rm(=d14) that is a combination of components of |V1p-V2p|, to therebydetermine a first motion vector vm=V1p-V2p, then adds the first motionvector vm to the first character vector V1(φ_(b))=V11(φ_(a))+V12(φ_(x),φ_(y) (and φ_(z))) of each first pixel Px(T) in the first minute piecePp(T) in concern to obtain for each first pixel Px(T) adimension-increased character vector V1(φ_(b), rm)=V11(φ_(a))+V12(φ_(x),φ_(y) (and (φ_(z)))+vm(rm) thereof.

The image segmentation circuit 23 segments the first picture Pc(T) intoM×N ( ×R) first regions (M, N, R=predetermined integers such that M<Mp,N<Np, R<Rp) each respectively composed of a cluster C1i (i=arbitraryinteger) of j (j=arbitrary integer) first pixels Px(T), so that thedimension-increased character vectors V1(φ_(b), rm) of the j firstpixels Px(T) are relatively vicinal to each other in terms of aeuclidean distance or L1 norm distance defined in the spatiotemporalfield and have the first location parameter vectors V12 thereof averagedto obtain a region location representative vector V1r representative ofa location (φ_(x), φ_(y) (and φ_(z))) of the first region C1i in concernin the spatiotemporal field.

The motion estimator 13 determines for each first region C1i acorresponding second region C2i in the second picture Pc(T-1), whichsecond region C2i is composed of j second pixels Px(T-1) such that thesecond character vectors V2 of the j pixels Px(T-1) or the second imageparameter vectors V21 thereof are each respectively relatively vicinalto an averaged or representative vector V1r of the first charactervectors V1 of the j first pixels Px(T) in the first region C1i or anaveraged or representative vector V11r of the first image parametervectors V11 thereof, respectively, in terms of the euclidean or L1 normdistance. Then, the estimator 13 calculates an average of the secondlocation parameter vectors V22 of the j second pixels Px(T-1) in thesecond region C2i to obtain a region location representative vector V2rrepresentative of a location (φ_(x), φ_(y) (and φ_(z))) of the secondregion C2i in the spatiotemporal field, and estimates a differencevector V1r-V2r between the region location representative vector V1r ofthe first region C1i and the region location representative vector V2rof the second region C2i, as a second relative motion RM (=D14) that isa combination of components of |Vlr-V2r|, to thereby determine a secondmotion vector VM=V1r-V2r.

The motion compensator 14 responds to the result RS of estimation tosegment a blank fourth picture Pc'(T) into M×N ( ×R) regions eachrespectively composed of a cluster C3i of j fourth pixels Px'(T) eachrespectively having a fourth character vector V4 equivalent to a vectorsum of a fourth image parameter vector V41 (=0) representative of a nullset of image data and a fourth location parameter vector V42 equivalentto the first location parameter vector V12 of one of the j pixels Px(T)in a corresponding first region C1i of the first picture Pc(T), and tothe motion vector VM associated with the first region C1i to add areverse vector of the motion vector VM to the fourth location parametervector V42 of a respective one of the j fourth pixels Px'(T) to therebydetermine a corresponding third pixel Px'(T-1) in the third picturePc'(T-1), and motion-compensates the third picture Pc'(T-1) by addingthe motion vector VM to the third location parameter vector V32 of thethird vector V3 of the corresponding third pixel Px'(T-1) or alternatelyby adding the third image parameter vector V31 of this third pixelPx'(T-1) to the fourth image parameter vector V41 of the respectivefourth pixel Px'(T), so that the respective fourth pixel Px'(T) has adata-added fourth vector such thatV4=V3+VM=(V31+V32)+VM=V31+(V42-VM)+VM=V31+V42. A set of such fourthvectors is linearly inverse converted to obtain the predicted pictureP14.

The first mode will be described below.

The first mode includes three fundamental steps. A first stepimage-segments the current picture P11 into a number of first regions. Asecond step determines for each first region a corresponding secondregion in the previous picture P13. A third step estimates for eachfirst region a relative motion RM thereof to the corresponding secondregion to determine a motion vector VM representative of the relativemotion RM.

In other words, the mode control switch 23b is turned off so that theimage segmentation circuit 23 directly image-segments the currentpicture P11 into a number of first regions, and outputs a set of dataD23 on a result RS of the segmentation to the motion estimator 13,motion compensator 14 and encoding multiplexer 20. Then, the motionestimator 13 determines for each first region a corresponding secondregion in the previous picture P13, estimates a relative motion RMtherebetween to determine the motion vector VM, and outputs a set ofdata D14 on the motion vector VM to the motion compensator 14 andencoding multiplexer 20. Thereafter, the first mode proceeds in the samemanner as the fourth mode.

In the first mode, therefore, the image segmentation circuit 23 segmentsthe first picture Pc(T) in the spatiotemporal field into M×N ( ×R) firstregions each respectively composed of a cluster C1i of j first pixelsPx(T), so that the character vectors V1(φ_(b)) of the j first pixelsPx(T) are relatively vicinal to each other in terms of a euclideandistance or L1 norm distance and have the first location parametervectors V12 thereof averaged to obtain a region location representativevector V1r representative of a location (φ_(x), φ_(y) (and φ_(z))) ofthe first region C1i in concern. The motion estimator 13 determines foreach first region C1i a corresponding second region C2i in the secondpicture Pc(T-1), which second region C2i is composed of j second pixelsPx(T-1) such that the second character vectors V2 of the j pixelsPx(T-1) or the second image parameter vectors V21 thereof are eachrespectively relatively vicinal to an averaged or representative vectorV1r of the first character vectors V1 of the j first pixels Px(T) in thefirst region C1i or an averaged or representative vector V11r of thefirst image parameter vectors V11 thereof, respectively, in terms of theeuclidean or L1 norm distance. Then, the estimator 13 calculates anaverage of the second location parameter vectors V22 of the j secondpixels Px(T-1) in the second region C2i to obtain a region locationrepresentative vector V2r representative of a location (φ_(x), φ_(y)(and φ_(z))) of the second region C2i in the spatiotemporal field, andestimates a difference vector V1r-V2r between the region locationrepresentative vector V1r of the first region C1i and the regionlocation representative vector V2r of the second region C2i, as arelative motion RM (=D14) that is a combination of components of|V1r-V2r|, to thereby determine a motion vector VM=V1r-V2r.

The second mode will be described below.

The second mode includes five or six fundamental steps. A first stepdivides the current picture P11 into a predetermined number of currentminute pieces each respectively consisting of one or more current pixelsPx. A second step may divide the previous picture P13 into an identicalnumber of previous minute pieces each respectively consisting of one ormore previous pixels Px. A third step determines for each current minutepiece a corresponding previous minute piece. A fourth step estimates foreach current minute piece a relative motion rm thereof to thecorresponding previous minute piece. A fifth step provides each pixel inevery current minute piece with a combination of motion data d14representative of the relative motion rm, as additional character data.A sixth step image-segments the current picture P11 into a number offirst regions with respect to the combination of image data, thecombination of address data and the combination of motion data.

In other words, first, the motion estimator 13 divides the currentpicture P11 into a predetermined number of current minute pieces eachrespectively consisting of one or more current pixels Px, and in somecase may divide the previous picture P13 into an identical number ofprevious minute pieces each respectively consisting of one or moreprevious pixels Px. The motion estimator 13 determines for each currentminute piece a corresponding previous minute piece, estimates a relativemotion rm therebetween so that each current pixel Px is additionallydefined by a combination of motion data d14 representative of therelative motion rm, and outputs the motion data d14 via the mode controlswitch 23b to the image segmentation circuit 23 together with associatedaddress data.

The image segmentation circuit 23 image-segments the current picture P11into a number of first regions with respect to the combination of imagedata, the combination of address data and the combination of motiondata, and outputs a result RS of the segmentation to the motioncompensator 14 and encoding multiplexer 20.

The result RS of segmentation is output also to the motion estimator 13,which however does not respond thereto to perform any reestimation inthis second mode. Accordingly, the estimator outputs a set of data D14representative of a zero motion vector VM=0.

The motion compensator 14 motion compensates the local decoded pictureP12 by using the result RS of segmentation and the zero motion vector,so that the picture P12 is image-segmented and image-mosaicked to obtaina predicted picture P14 of the current picture P11, i.e., all pixels inthe picture P12 are clustered into a number of regions and respectivepixels in each region are equalized thereamong to have a combination ofcomponentwise averaged or region-representative image data. Thereafter,like operations to the fourth mode follow, subject to VM=0 and animage-mosaicking operation of the motion compensator 33 (at the decodingend).

In a modified case of the second mode, an exclusive motion compensatoror image-mosaicking circuit may be incorporated in theimage-segmentation circuit 23 so that the image-segmentation circuit 23may further serve for image-mosaicking the current picture P11 todirectly obtain an improved prediction picture of the current pictureP11, for a compression purpose. In that case, a set of data D23 on theprediction picture may be output as a result RS of segmentation from theimage-mosaicking circuit to the subtractor 15, where the predictionpicture RS may be subtracted from the current picture P11 to obtain aprediction error picture P15 to be processed in a similar manner to thefourth mode, as well as to the encoding multiplexer 20, where the dataD23 may be encoded into a sequence of compressed codes, which codes maybe multiplexed and transferred to the decoding multiplexer 31, wherethey may be demultiplexed and decoded into a set of decompressed dataD36 directly representative of a coded prediction picture RS, which maybe added to a decoded error picture P31 (i.e. a picture decoded by thedecompression circuit 32 to represent the prediction error picture P15)to obtain a decoded picture P33 of the current picture P11. Likeoperations may be employed to obtain a local decoded picture P17 of thecurrent picture P11, by applying the data 23 (in place of the data D15)to the adder 18.

In the modified case, therefore, the motion compensators 14 and 33 (atboth encoding and decoding ends) may be cut off together with theirassociated lines so that in the encoder 10, an image mosaicking circuitof the image segmentation circuit 23 may be connected to the subtractor15, the adder 18 and the encoding multiplexer 20 and, in the decoder 30,the decoding demultiplexer 31 may be connected directly to the adder 34as well as to the decompression circuit 32.

In the second mode, like the fourth mode, the motion estimator 13divides the first picture Pc(T) in the spatiotemporal field into Mp×Np(×Rp) first minute pieces Pp(T) each respectively consisting of one ormore first pixels Px(T) having the first location parameter vectors V12thereof averaged to obtain a piece location representative vector V1prepresentative of a location (φ_(x), φ_(y) (and φ_(z))) of the firstminute piece Pp(T) in the spatiotemporal field, and in some case maydivide the second picture Pc(T-1) into Mp×Np (×Rp) second minute piecesPp(T-1) each respectively consisting of one or more second pixelsPx(T-1) having the second location parameter vectors V22 thereofaveraged to obtain a piece location representative vector V2prepresentative of a location (φ_(x), φ_(y) (and φ_(z))) of the secondminute piece Pp(T-1) in the spatiotemporal field. The estimator 13determines for each first minute piece Pp(T) a corresponding secondminute piece Pp(T-1), and estimates a difference vector V1p-V2ptherebetween, as a first relative motion rm (=d14) that is a combinationof components of |V1p-V2p|, to thereby determine a first motion vectorvm=V1p-V2p, then adds the first motion vector vm to the first charactervector V1(φ_(b))=V11(φ_(a))+V12(φ_(x), φ_(y) (and φ_(z))) of each firstpixel Px(T) in the first minute piece Pp(T) in concern to obtain foreach first pixel Px(T) a dimension-increased character vector V1(φ_(b),rm)=V11(φ_(a))+V12(φ_(x), φ_(y) (and φ_(z)))+vm(rm) thereof.

The image segmentation circuit 23 segments the first picture Pc(T) intoM×N ( ×R) first regions each respectively composed of a cluster C1i of jfirst pixels Px(T), so that the dimension-increased character vectorsV1(φ_(b), rm) of the j first pixels Px(T) are relatively vicinal to eachother in terms of a euclidean distance or L1 norm distance defined inthe spatiotemporal field and have the first location parameter vectorsV12 thereof averaged to obtain a region location representative vectorV1r representative of a location (φ_(x), φ_(y) (and φ_(z))) of the firstregion C1i in concern in the spatiotemporal field.

The third mode will be described below.

The third mode includes six fundamental steps. A first stepimage-segments the current picture P11 into a predetermined number ofcurrent regions. A second step divides each current region into avariable number of sub-regions each respectively consisting of one ormore current pixels. A third step determines for each sub-region inevery current region a corresponding small region in the previouspicture P13. A fourth step estimates for each sub-region a relativemotion rm thereof to the corresponding small region. A fifth stepprovides each pixel in each sub-region of every current region acombination of motion data d14 representative of the relative motion rm.A sixth step resegments the current picture P11 into a number of firstregions with respect to the combination of image data, the combinationof address data and the combination of motion data.

In other words, first, the image segmentation circuit 23 image-segmentsthe current picture P11 into a predetermined number of current regions.A result of this segmentation is output from the circuit 23 to themotion estimator 13. The motion estimator 13 divides each current regioninto a variable number of sub-regions each respectively consisting ofone or more current pixels Px, determines for each thereof acorresponding small region in the previous picture P13, estimates arelative motion rm therebetween, and additionally defines each currentpixel Px by a combination of motion data d14 representative of therelative motion rm. A set of those motion data d14 is input from theestimator 13 via the mode control switch 23b to the image segmentationcircuit 23.

Then, the image segmentation circuit 23 resegments the current pictureP11 into a number of first regions with respect to the combination ofimage data, the combination of address data and the combination ofmotion data d14. Thereafter, like operations to the second mode follows,including the modified case thereof.

Therefore, in a modified case of this third mode, the motion pictureprocessing system 1 may include a resegmentor (as part of the circuit23) for image-mosaicking the current picture P11 to obtain a predictionpicture thereof as a result RS of estimation so that respective currentpixels Px associated with a respective one of the first regions have acombination of region-representative image data in place of thecombinations of image data thereof, a subtractor 15 for subtracting theprediction picture RS from the current picture P11 to obtain aprediction error picture P15, a compressing coder 16 for coding theprediction picture RS and the prediction error picture P15 in acompressing manner into a first set of coded data and a second set ofcoded data, respectively, a decompressing decoder 17 for decoding thefirst set of coded data and the second set of coded data in adecompressing manner into a local decoded prediction picture RS and alocal decoded error picture P16, respectively, and an adder 18 foradding the local decoded error picture P16 to the local decodedprediction picture RS to obtain a local decoded picture P17. As a matterof course, the prediction picture RS may be directly (i.e. without acoding-decoding operation) added to the local decoded error picture P16to obtain a local decoded picture P17.

Further, in a modified case of the third mode, the encoder 10 maycomprise a segmentor 23 for image-segmenting the current picture P11into a predetermined number of current regions, a motion estimator 13for dividing each current region into a variable number of sub-regionseach respectively consisting of one or more current pixels Px,determining for each sub-region of each current region a correspondingsmall region in the previous picture P13, estimating a relative motionrm therebetween and additionally defining each current pixel by acombination of motion data d14 representative of the relative motion rm,a resegmentor 23 for resegmenting the current picture P11 into a numberof first regions with respect to the combination of image data, thecombination of address data and the combination of motion data d14 andimage-mosaicking the current picture P11 to obtain a prediction pictureRS thereof so that respective current pixels associated with arespective one of the first regions have a combination ofregion-representative image data in place of the combinations of imagedata thereof, a subtractor 16 for subtracting the prediction picture RSfrom the current picture P11 to obtain a prediction error picture P15,and a encoding multiplexer 16+20 for coding the prediction picture RSand the prediction error picture P15 in a compressing manner into asequence of first codes and a sequence of second codes, respectively,and multiplexing the sequence of first codes and the sequence of secondcodes to obtain a multiplexed signal C11.

The decoder 30 may comprise a decoding demultiplexer 31+32 fordemultiplexing the multiplexed signal C11 into a combination of asequence of third codes corresponding to the sequence of first codes anda seqeunce of fourth codes corresponding to the sequence of second codesand decoding the sequence of third codes and the sequence of fourthcodes in a decompressing manner into a set of data D36 representative ofa decoded prediction picture RS and a set of data D33 representative ofa decoded error picture P31, respectively, and an adder 34 for addingthe decoded error picture P31 to the decoded prediction picture RS toobtain a decoded picture P33 corresponding to the current picture P11.

In the third mode, the image segmentation circuit 23 segments the firstpicture Pc(T) in the spatiotemporal field into M'×N' (×R') (M', N',R'=predetermined integers such that M'<M, N'<N, R'<R) initial regionseach respectively composed of a cluster C1i' of k (k=arbitrary integersuch that k>j) first pixels Px(T), so that the character vectorsV1(φ_(b)) of the k first pixels Px(T) are relatively vicinal to eachother in terms of a euclidean distance or L1 norm distance.

The motion estimator 13 divides each region C1i' of the first picturePc(T) into (Ms/M')×(Ns/N') {×(Rs/R')} (Ms, Ns, Rs=arbitrary integerssuch that Ms>M, Ns>N, Rs>R) sub-regions Ps(T) each respectivelyconsisting of one or more first pixels Px(T) having the first locationparameter vectors V12 thereof averaged to obtain a sub-region locationrepresentative vector V1s representative of a location (φ_(x), φ_(y)(and φ_(z))) of the sub-region Ps(T) in the spatiotemporal field, and insome case may divide the second picture Pc(T-1) into Ms×Ns (×Rs) smallregions Ps(T-1) each respectively consisting of one or more secondpixels Px(T-1) having the second location parameter vectors V22 thereofaveraged to obtain a small-region location representative vector V2srepresentative of a location (φ_(x), φ_(y) (and φ_(z))) of the smallregion Ps(T-1) in the spatiotemporal field. The estimator 13 determinesfor each sub-region Ps(T) a corresponding small region Ps(T-1), andestimates a difference vector V1s-V2s therebetween, as a relative motionrm (=d14) that is a combination of components of |V1s-V2s|, to therebydetermine a motion vector vm=V1s-V2s, then adds the motion vector vm tothe first character vector V1(φ_(b))=V11(φ_(a))+V12(φ_(x), φ_(y) (andφ_(z))) of each first pixel Px(T) in the sub-region Ps(T) in concern toobtain for each first pixel Px(T) a dimension-increased character vectorV1(φ_(b), rm)=V11(φ_(a))+V12(φ_(x), φ_(y) (and φ_(z)))+vm(rm) thereof.

Then, the image segmentation circuit 23 segments the first picture Pc(T)into M×N(×R) first regions each respectively composed of a cluster C1iof j first pixels Px(T), so that the dimension-increased charactervectors V1(φ_(b), rm) of the j first pixels Px(T) are relatively vicinalto each other in terms of a euclidean distance or L1 norm distancedefined in the spatiotemporal field and have the first locationparameter vectors Vl2 thereof averaged to obtain a region locationrepresentative vector V1r representative of a location (φ_(x), φ_(y)(and φ_(z))) of the first region C1i in concern in the spatiotemporalfield.

The foregoing first to fourth modes may be selected in accordance with aresult RS of estimation of a past input picture.

Incidentally, in a modified case of the first or fourth mode, a previouspicture P13 may be image-segmented with respect to image and locationdata and/or motion data of pixels thereof to provide a result RS ofsegmentation and each image-segmented region of the picture P13 may bemotion-estimated relative to the current picture P11 to obtain a motionvector VM, before a motion compensation of the local decoded picture P12in which each region defined in accordance with the result RS ofsegmentation may have respective pixels thereof displaced in dependenceon the motion vector VM together with their image data to obtain aprediction picture P14 of the current picture P11.

Further, in a normal operation flow of each of the second to fourthmodes, the previous picture P13 is not divided into minute pieces norsmall regions, before the motion estimator 13 determines for each minutepiece or sub-region of the current picture P11 a corresponding minutepiece or small region in the previous picture P13 such that respectivepixels Px(T-1) in this minute piece or small region have their charactervectors V2 or image parameter vectors V21 each respectively relativelyvicinal to an averaged or representative vector of respective charactervectors V1 or image parameter vectors V11r of pixels Px(T) in thatminute piece or sub-region.

The spatiotemporal field φ-T and its relationship with pixel characterparameters φ_(b) will be described.

As used herein, the term "character" or "pixel character" means any andall features that define a pixel to be processed in a method or systemaccording to the invention and are sufficiently significant to berecognized in the method or by the system. For example, in theembodiments described, each pixel is defined by a combination of imagedata (i.e. a monochromatic luminance signal, three color data R-G-B orthree luminance and chrominance data Y-Cr-Cb) and a combination oflocation data (i.e. two- or three-element address matrix or spatialcoordinates X-Y(-Z)) with a sufficient significance when input to thesystem, and is additionally defined by a combination of motion data(i.e. vx-vy(-vz) or Vx-Vy(-Vz)) with a sufficient significance in theway of an associated process. In a field of a virtual image orpara-experience, a pixel may be additionally defined by a combination ofincentive data such as on a scent, pressure, temperature or brain wave.

The term "parameter" or "character parameter" means a parameter definedin the method or system, to represent a degree, level or magnitude ofthe significance of a character.

Accordingly, when a pixel Px is defined by a set φ of characterparameters φ_(bi), where i is an arbitrary integer as suffix such that1≦i≦p (predetermined integer), any pair of elements φ_(bn) and φ_(bm) ofthe set φ have an intersection φ_(bn) ∩φ_(bm) therebetween alwaysequivalent to a null set φ if n≠m, where n and m are integers such that1≦n≦p and 1≦m≦p.

As shown in FIG. 17, any pixel Px, that is fully defined by a unique setφ={φ_(bi) (T)}, and hence any cluster Ci or small region in a pictureframe Fp of a current picture Pc(T) is congruently mapped into animaginary frame Fc in the spatial field X-Y(-Z), wherefrom it islinearly mapped into the spatiotemporal field φ-T which is a vectorfield defined by a coordinate system φ_(b1) -φ_(b2) -. . . -φ_(bi) -. .. -φ_(bp) -T. Thus, letting D_(iT) be a parameter value of φ_(bi) (T)and e_(i) and e_(T) be unit vectors along φ_(bi) -axis and T-axis,respectively, the set φ={φ_(bi) (T)} is uniquely mapped as a vector V(D_(iT),T)=D_(1T) ·e₁ +D_(2T) ·e₂ +. . .+D_(iT) ·e_(i) +. . . +D_(pT)·e_(p) +T·e_(T). The cluster Ci(T) or small region is uniquely mapped asa set of such vectors {V (D_(iT),T)}. Letting _(bi) =x, _(b2) =y and_(b3) =z, the vector V (D_(iT),T) has a spatial location thereof definedby a location vector V12 with a combination of spatial locationcomponents D_(1T), D_(2T) and D_(3T).

Likewise, any pixel Px, that is fully defined by a unique set φ={φ_(bi)(T-1)}, and hence any cluster Ci(T-1) or small region in a picture frameFp of a previous picture Pc(T-1) is congruently mapped into theimaginary frame Fc in the spatial field X-Y(-Z), wherefrom it islinearly mapped into the spatiotemporal field φ-T, where the setφ={_(bi) (T-1)} is uniquely mapped as a vector V (D_(i)(T-1), T-1)=D₁(T-1) ·e₁ +D₂ (T-1) ·e₂ +. . . +D_(i) (T-1) e_(i) +. . . +D_(p) (T-1)·e_(p) +(T-1)·e_(T). The vector V (D_(i)(T-1),T-1) also has a spatiallocation thereof defined by a location vector V22 with a combination ofspatial location components D₁ (T-1), D₂ (T-1) and D₃ (T-1). The clusterCi(T-1) or small region is also uniquely mapped as a set of such vectors{V (D_(i)(T-1), T-1)}.

In the motion estimation, letting V1r be a representative vector(D_(1Tr), D_(2Tr), D_(3Tr), D_(4Tr), . . . D_(1Tr), . . . D_(pTr),T) ofthe vector set {V (D_(iT),T)}, each mapped vector (as a i-th identifiedone) in a vicinity of a spatially co-located vector V (D_(1Tr), D_(2Tr),D_(3Tr), D₄ (T-1) a, . . . D_(i)(T-1) a, . . . D_(p)(T-1) a, T-1)(suffix a=arbitrary value) with the vector V1r is checked for acharacter difference vector δ1=(D_(4Tr) -D₄ (T-1)j)e₄ +. . . +(D_(1Tr)-D₁ (T-1)j)e_(i) +. . . +(D_(pTr) D_(p) (T-1) j)e_(p) therebetween to beminimal or relatively small in magnitude. If the cluster Ci(T-1) orsmall region therein is spatially displaced to constitute the clusterCi(T) or small region therein, the difference vector δ1 with respect toany and all pixels in the former will be confirmed to fall within aconforming magnitude range. For each conforming pixel to V1r, a spatiallocation difference vector Vd is calculated such that Vd=(D_(1Tr) ·e₁+D_(2Tr) ·e₂ +D_(3Tr) ·e₃)-V22, and is averaged among all conformingpixels to V1r to determine a motion vector VM or vm. As a result, eachconforming pixel Px(T) to V1r has an additional character parameterrepresentative of the motion vector VM or vm as partially illustrated ina X- φ₂ (G)-Vy space in FIG. 17.

While the present invention has been described with reference to theparticular illustrative embodiments, it is not to be restricted by thoseembodiments but only by the appended claims. It is to be appreciatedthat those skilled in the art can change or modify the embodimentswithout departing from the scope and spirit of the present invention.

What is claimed is:
 1. A method for processing a sequence of motion pictures, each motion picture comprising a plurality of picture elements, said method comprising the steps of:determining picture element information for each picture element in a current picture of said sequence of motion pictures, said picture element information including a plurality of components comprisinga signal value of said each picture element, a coordinate position of said each picture element, and a motion estimation vector of said each picture element, said motion estimation vector being determined based on a position of said picture element in said current picture and a position of at least one corresponding picture element in a previous picture to said current picture; image-segmenting said current picture in said sequence of motion pictures into a plurality of first regions, wherein the image-segmenting is dependent on each component of said picture element information for each of said plurality of picture elements of said current picture; determining for each of said plurality of first regions a corresponding second region in said previous picture to said current picture in said sequence of motion pictures; and estimating a relative motion between each of said plurality of first regions and said corresponding second region to determine a motion vector representative of said relative motion.
 2. A method according to claim 1, further comprising the steps of:motion-compensating a local decoded picture of said previous picture in dependence on said motion vector to obtain a predicted picture of said current picture; subtracting said predicted picture from said current picture to obtain a prediction error picture; coding said prediction error picture in a compressing manner into a set of coded data; decoding said set of coded data in a decompressing manner into a local decoded error picture; and adding said local decoded error picture to said predicted picture to obtain a local decoded picture of said current picture.
 3. A method according to claim 1, wherein said current picture comprises an arbitrary picture in said sequence of motion pictures, andwherein said previous picture comprises a local decoded picture of a picture previous to said arbitrary picture in said sequence of motion pictures.
 4. The method of claim 1, wherein a shape of each of said plurality of first regions is indeterminate prior to said image-segmenting step.
 5. The method of claim 1, wherein a shape of each of said plurality of first regions is dependent upon the picture element information for each of the plurality of picture elements in said current picture.
 6. A system for processing a sequence of motion pictures, each motion picture comprising a plurality of picture elements, said system comprising:picture element information determining means for determining picture element information for each picture element in a current picture of said sequence of motion pictures, said picture element information including a Plurality of components comprisinga signal value of said each picture element, a coordinate position of said each picture element, and a motion estimation vector of said each picture element, said motion estimation vector being determined based on a position of said picture element in said current picture and a position of at least one corresponding picture element in a previous picture to said current picture; image segmenter means for image-segmenting said current picture in said sequence of motion pictures into a plurality of first regions, wherein the image-segmenting is dependent on each component said picture element information for each of said plurality of picture elements of said current picture; and motion estimator means for determining for each of said plurality of first regions a corresponding second region in a previous picture to said current picture in said sequence of motion pictures, and for estimating a relative motion between each of said plurality of first regions and said corresponding second region to determine a motion vector representative of said relative motion.
 7. A system according to claim 6, further comprising:a motion compensator means for motion-compensating a local decoded picture of said previous picture in dependence on said motion vector to obtain a predicted picture of said current picture; a subtractor means for subtracting said predicted picture from said current picture to obtain a prediction error picture; a compressing coder means for coding said prediction error picture in a compressing manner into a set of coded data; a decompressing decoder means for decoding said set of coded data in a decompressing manner into a local decoded error picture; and an adder means for adding said restored error picture to said predicted picture to obtain a local decoded picture of said current picture.
 8. A system according to claim 6, wherein said current picture comprises an arbitrary picture in said sequence of motion pictures, andwherein said previous picture comprises a local decoded picture of a picture previous to said arbitrary picture in said sequence of motion pictures. 